{"id":24452,"date":"2020-04-09T14:09:50","date_gmt":"2020-04-09T14:09:50","guid":{"rendered":"https:\/\/xomultimedia.com\/blog\/?p=24452"},"modified":"2024-10-20T12:14:51","modified_gmt":"2024-10-20T12:14:51","slug":"robots-txt-kya-hai","status":"publish","type":"post","link":"https:\/\/xomultimedia.com\/blog\/robots-txt-kya-hai\/","title":{"rendered":"Robots.txt Aur SEO: Complete Guide"},"content":{"rendered":"\n

Robots.txt website pe sab se simplest files me se ek hai, iske sath hi ye un files me se bhi ek hai jaha pe hum sab se aasani ke sath kuch bhi garbari kar sakte hai. Yahan pe aap agar ek character ko bhi apni jagah se bejagah kar dete hai to ye aapke SEO ko destroy kar sakta hai, aur search engine ko aapke site ke important content ko access karne se rok sakta hai. Is article me hum aapko batayenge ke Robots.txt kya hai, Robots.txt kyu important hai?<\/p>\n\n\n\n

\"\"<\/figure><\/div>\n\n\n\n

Robots.txt kya hai?<\/strong><\/h3>\n\n\n\n

Ye ek aisa file hai jo search engine spider ko batata hai ke use kisi website ke kis pages ya section ko crawl nahi karna hai. Zyadatar search engine (Google, Bing aur Yahoo) Robots.txt requests ko pahchahnte hai aur use accept bhi karte hai.<\/p>\n\n\n\n

Robots.txt file ko webmasters web robots (search engine robots) ko instruct karne ke liye create karte hai ke wo unke website ke pages ko kaise crawl kare. <\/p>\n\n\n\n

Practically robots.txt files indicate karta hai ke certain user agents (web-crawling software) website ke kin kin parts crawl kar sakta hai aur kin parts ko nahi. <\/p>\n\n\n\n

Robots.txt Important Kyu Hai?<\/strong><\/h3>\n\n\n\n

Bahut sari sites aise hoti hai jinhe robots.txt file ki zaroorat nahi hoti hai.<\/p>\n\n\n\n

Aisa isliye hai ke Google apke site ke sabhi important pages ko find aur index kar sakta hai. Aur wo automatically un pages ko index nahi karta hai jo ya to important na ho ya kisis dusre pages ka duplicate version ho.<\/p>\n\n\n\n

3 aise main reasons hai jinki wajah se aap robot.txt file ka use karna chahte hai.<\/p>\n\n\n\n

Block Non-Public Pages:<\/strong> kabhi kabhi hamare site pe aise kuch pages hote hai jinhe hum index karna nahi chate hai. Example ke liye aapke page ka staging version ya login page. In pages ko aapko rakhna zaroori hai, par aap nahi chahenge ke log ispe koi randopeople land kare. <\/p>\n\n\n\n

Aur isis tarah ke cases ke liye aapko robot.txt file ki zaroorat hai, taki aap search engine crawler ko apne in pges ko crawl karne se rok sake.<\/p>\n\n\n\n

Maximize Crawl Budget:<\/strong> Agar aapko apne sabhi pages ko index karne me problem aa rahi hai to aapko crawl budget<\/a> ki problem ho sakti hai. Unimportant pages ko robot.txt ki help se block kar ke, Googlebot aapke crawl budget ke un pages pe zyada spend kar sakta hai jo actuall important ho.<\/p>\n\n\n\n

Prevent Indexing of Resources:<\/strong> Robots.tx ki tarah meta directives bhi aapko pages ko index karne se prevent karta hai. Halaki meta directives multimedia resources, jaise PDFs aur images ke sath kaam nahi karta hai. Aur yahi wo jagah jub robots.txt ki zaroorat hoti hai.<\/p>\n\n\n\n

Robots.tx search engine ko aapki site ke kisi specific pages ko crawl karne se rokta hai. Aap search console me dekh sakte hai ke aapke kitne pages indexed hai.<\/p>\n\n\n\n

Agar ye number us pages ke number se match karta hai jinhe aap index karna chahte hai, to aapko Robots.tx file ke liye bother hone ki zaroorat nahi hai.<\/p>\n\n\n\n

Aur agar ye number aapke expected pages ke number se zyada hai to aapko apne website ke liye Robots.tx file ko create karne ki zaroorat hai.<\/p>\n\n\n\n

Best Practices<\/strong><\/h3>\n\n\n\n

Ab tak hum jaan chuke hai ke Robots.txt file kya hota hai aur ye important kyu hai. Ab hum aapko kuch aise best practices ke bare me batayenge jinhe aapko follow karna hai.<\/p>\n\n\n\n

Also Read: 10 Most Important Parts of SEO<\/a><\/p>\n\n\n\n

Create a Robots.txt File<\/strong><\/p>\n\n\n\n

Aapka pahla step robots.txt file create karna hai.<\/p>\n\n\n\n

Aap apna robots.txt file kisi bhi plain text editor ka use kar ke bana sakte hai.<\/p>\n\n\n\n

Agar aapke pass pahle se robots.txt file hai to aap uske text ko delete kar de. Aapko sabse pahle user-agent term ko setup karna hoga.<\/p>\n\n\n\n

Sabs pahle aapko robots.txt file me use kiye gaye kuch terms ko samajhna hoga, in terms ke liye aap Google ke define kiye gaye in terms <\/a>ko dekh sakte hai.<\/p>\n\n\n\n

\"Robots.txt<\/a><\/figure><\/div>\n\n\n\n

Ab aapko ek simple robot.txt file ko setup karna hai.<\/p>\n\n\n\n

User-agent: X<\/em>
Disallow: Y<\/em><\/p>\n\n\n\n

User-agent wo specific bot jo ise follow karega. Aur \u201cdisallow\u201d ke baad aane wale sabhi chize aise pages ya sections hai jise aap block karna chahte hai.<\/p>\n\n\n\n

Example: <\/p>\n\n\n\n

User-agent: googlebot<\/em>
Disallow: \/images<\/em><\/p>\n\n\n\n

Ye rule Googlebot ko aapke website ke us image folder ko index karnese mana karega.<\/p>\n\n\n\n

Agar aap aisa chahte hai ke sabhi web robots ise follow kare to aap asterisk (*) ka use kar sakte hai.<\/p>\n\n\n\n

Example:<\/p>\n\n\n\n

User-agent: *
Disallow: \/images<\/p>\n\n\n\n

Asterisk (*) sabhi spider ko aapke images folder ko crawl karne se rok dega.<\/p>\n\n\n\n

Ye robots.txt file ko use karne ke kayi tareeqo me se ek hai.<\/p>\n\n\n\n

Robots.txt File Ko Easy to Find Banaye<\/strong><\/h4>\n\n\n\n

Apne root.txt file ko banane ke baapko ise live karna hoga.<\/p>\n\n\n\n

Aap technically robots.txt file ko apne site ke main directory me kahi pe bhi place kar sakte hai. Lekin hamara recommendation hai ke aap ise is yaha place kare:<\/p>\n\n\n\n

https:\/\/example.com\/robots.txt<\/p>\n\n\n\n

Aapka robots.txt file case sensetive hai isliye aapko apne file name me lowercase “r” ka use kare.<\/p>\n\n\n\n

Errors Aur Mistakes Ko Check Kare<\/strong><\/h4>\n\n\n\n

Aapke robots.txt file ka correctly setup hona bahut zaroori hia. Agar aap yaha pe ek bhi mistake karenge to ho sakta hai ke aapka poora site de-index ho jaye.<\/p>\n\n\n\n

Fortunately, aapko is baat ke liye pareshan hone ki zaroorat nahi ai ke aapka code sahi set up hua hai, kyuki Google ke nifty Robots Testing Tool<\/a> ka use aap kar sakte hai.<\/p>\n\n\n\n