The Barbarians is a team made up of players from all over the world, and is generally considered a World XV, with national teams seeing it as a game to start against uncapped players.
The RFU says several players left the team hotel on Wednesday night, without revealing their whereabouts and have now been separated from the rest of the squad bubble.
Sky Sports News understands that 12 players have left the hotel.
Even though there is another round of testing for the game, the RFU said there could be no guarantee that they were not infectious in the lead-up to the game and that they had been banned from Game.
A decision regarding the game will be made on Friday at 3 p.m.
The RFU statement read: “To protect the safety of other participants, these players cannot be allowed to stay with the Barbarians team and play in Sunday’s game.
“If a sufficient number of new players with an appropriate testing history and who meet the standards of the Code of Conduct can be brought into the Barbarians’ camp, then the game will continue.
“The RFU has scheduled the Barbarians Quilter Cup match for performance purposes to provide a warm-up match prior to the postponed Guinness Six Nations match Italy v England.
“However, this will not compromise the safety of the players. ”
The statement continued: “Barbarian FC have expressed their extreme disappointment with the conduct of the players involved and have taken all necessary measures to ensure the safety of the players and staff within the bubble.
“The club is working closely with the RFU to determine if the Quilter Cup can go ahead.
“The RFU will provide a status update by 3pm on Friday, October 23.
“No details on who will or will not be part of the revised Barbarian squad will be announced until the match is confirmed. “