Knowledge versus conjecture

وَقَالُوا۟ لَوْ شَآءَ ٱلرَّحْمَـٰنُ مَا عَبَدْنَـٰهُم ۗ مَّا لَهُم بِذَٰلِكَ مِنْ عِلْمٍ ۖ إِنْ هُمْ إِلَّا يَخْرُصُونَ

And they said, "If the Most Merciful had willed, we would not have worshipped them." They have of that no knowledge. They are not but misjudging.