Robot.txt is a file to the direction to the web crawler how to crawl their page or website. Robot.txt is one of the protocol to regulate how the crawler, crawling your content, pages, and shows the users. This also instructed the robots how search engine treats the links.
Robot.txt files instruct where the user agent can or can not crawl the website. This indicates allowing and disallowing to crawl your webpage.
-The file must be placed your top-level directory
-This file named robot.txt
-The robot.txt files must be publicly available to see.
Watch this video and learn how to robot.txt files place in a right place.
robots.txt- রোবটস টেক্সট ফাইলটি এসইও এর জন্যে কাজে লাগানোর ক্ষেত্রে এই ভিডিও টিউটোরিয়াল থেকে আপনারা যা যা শিখতে পারবেন ঃ
– How to write perfect robots.txt file.
– How to allow or disallow specific folder or a directory of a website.
– How to block similar type of pages to search engines.
– What is crawl delay and how to write it down on a robots.txt file.
Subscribe me on YouTube for latest Bangla tutorials. Choose HD in YT settings for better viewing. Thank You. 🙂
Vaia age jantam metter ta but kichu kichu aj notun shikhlam.
Thanks for this helpful video! 🙂
Keep learning 🙂
Hello Shamim sir, Thank you very very much I’ve learn lots of things online from you still learning from you, please sir keep rocking ,
You’re my Virtual Guru, sir please take a Tour of my Website https://www.itguruweb.com
Thank You 🙂
Thanks for your nice video article. I have watched most of the video to learn SEO at home. I like your videos. My website robots.txt file shown as follow-
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
sitemap: https://www.pixelzcenter.com/sitemap.xml
Is it ok or not? Awaiting for your kind response.
Why you are allowing your WP Admin directory? Is it necessary? If not, edit your robots.txt file and paste the code below –
User-agent: *
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /wp-includes/js
Disallow: /trackback
sitemap: https://www.pixelzcenter.com/sitemap.xml
Please help me, I have uploaded with the code which you mention in comment.
but when I open my site http://www.example.com/robots.txt. then its comes like this.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
is it okay?
Okay…but why did you allow your wp-admin on the second line?
Make it disallowed as well.
Sir tag and category ki disallow kora valo
ha tag and category disallow kore rakhben.