Web Automation with Python Selenium

1. What is selenium: selenium framework. Ultimate browsers, that's where browses. So that's right. We can completely control a Web browser from codes. So this guy is just a lot of superpowers. We can make automated testing, Web browsing and Web scraping on the weapon. More so. The Cellini in framework and controlled many browsers, including Firefox, Google Chrome, Internet Explorer, Safari Opara and tons of other Web browsers. 2. Webdriver: we need to install fightin in the Cellini in Web Driver. A selenium. What driver? A little bit. All at first. It accepts CO months, and it sends them to the browser. So that's like how Biden communicates. Delete my browser, where drivers can be found on the website. Selenia Hike, you dork. So once you greats your program and run it, it will start a Web rose. So if you used the grown driver, it will start Google Chrome. If you used a gecko driver, it starts Mozilla Firefox. Or, if you want, just want to completely hide what the browser is doing. You can use a Web driver called fencing Js, so after installing the browser will do anything you writing coat. Whether that's clicking on buttons, clicking on listen, books, typing in fields, clicking on images or whatever he wants it to do, you can completely control the Web browser from coats 3. Get Text: we can automatically get text from any weapons from any website or any weapon type imports time and in doing party Web driver then will initialize the Web driver, get your webpage, wait for a little bit and quit. So we end up in this program, then save it and run it so I could save it to example. Look, Brian's run. It's using on point and example that by So if you're sitting, if your Web drivers installed correctly, this will just open the Web browser. Then we changed the he met its driver. Don't get so the get methods we change now. Presti of 12 keys to open chrome developer tools. We need to get the actual exports, and she's like a unique identifier for two text for elements on the weapons so breast you have 12 key to open chrome developed ALS and Presti Small arrow and then simply point on the text or it's his views. You see one line off the coast is highlighted in blue, so right most click on debts and copy copy exports. You'll see this expert when you pay sitting note butter another program and just keep it open for now, because this is the unique identity Fire off this text. So then we go back to work coat and we after these lines. So we get the the element and then it's texting we simply printed. Now pay attention. We copy the expert from the browser, and now we're using the methods find element by exports. So this is how these two or intertwines 4. Automate Search: in this lecture will make a Web browser do. Automated search, in short, will make a Web browser, open the website and do a natural search awesome type of these coats. Then we want to select to search books. So every thing you see in a webpage has a expert what it's called. So to get to the unique, unique identifier for search books and we start grown and will open chrome developer tools to get the unique idea. Ah, or expert Presti F 12 key to open the chrome developer tools. And then you see small pointer. So click on it small arrow and then click on the text books. So then, on the right or the bottom, he'll see the one line highlight it, which shows the exports right click on that line ends press copy copy experts, and we saved the experts toe fell. So then we need to update our code because so far it only updates only opens a website. So first will get the elements using the methods find element by exports so that will gets the elements in the pipes and coach, and then we'll need to send keys. So there's a method called send keys. So we have these two lines and safe there foul and run the program. You'll see it opens the website and starts typing in the search box, but so far it does not actually search. So that's because the computer didn't press the enter key yet. So we had a new import and then at another line savings, run the program until then, started program. Visit the website type the extent press enter so all the search will appear automatically. 5. Links: So where pay she's defined in HTML language guage. So single link Its return is this and we can use this these tax to actually get the content that we want. So in this case, you want to get links. So we type these lines, of course. So find elements, my export. And then we use t h ref. And then we'd have a list of links to show each one of them. We type these lines so that will prints every list in the length. So the focus will be this now save it and ran its and you'll see a list of links on this websites. 6. Download images: selenium can be used to download images from a weapons show. Any weapons, you can download all of the images. That's what we're going to do in this election. So we take any. Any websites can be an image shines a product page or whatever any weapon she wants. So first we're going to open a weapons with one or more images. For example, in this one wheel, take forum threads containing scree shows. Then we want to start the default program, creating the driver hands, opening weapons and the weapon shorts Actually, low t the driver should actually open the link. So safe this program and to run it. We fightin, you'll see it open Sea webpage and Loach D No seepage. Then we have these lines. Eso we actually used the tech I am G, which is the image tech, and we use it using he finds elements my tech method so that will simply get all the texts that contain image for these letters and specifically from the tech. We want the source because the sorts of sexual linked to the image so that's always defined in the all of the images. So are these lines and save and run the program. After a few seconds. This will prints all the images links to the screen. So it isn't just the links, but not actually the images yet. And we can download all these images by combining it with the program. W gets so we make a directory for the images to be felt safety. Let's call it I am G. When we go back to the Colts and on top off the foul, we're going to call the program. So as imports always, then we change the line, this line to the line. In short, his image to get entry with source. So actually safety the link in a terrible got image shores. And then we'll have this land below Mr System W gets so safety image. So that's in a loop, so it will simply downloads images one by one. So if you saved it and run its then after a while, all the images on the Web page download it into the directory I m. G. So you can use this program went anywhere Page that contains images 7. Login: you can automate Loggins on webpages, so there's many Web perhaps around the weapon. It's you you need to log in like you need to have your email passwords. Then you have an account on the webpage threat so you can automate his loving process. So I just picked looking page off. Stack. Overflow does come, which is this hero, so you should go, and it simply has an email fields possible field loving button. So this will work. This technique will work with any webpage it has lovin. So whether that's email, social network or forum like stack overflow or really anything, and you can type those things automatically So the email field, the password fields and the Logan feels now to do so in et ideas off the field. If the ideas are not available, you can use experts, but in this case we have ideas. And so start your grown developer tools or Firefox developer tools by pressing the F 12 key and simply collecting aero hair and points on the elements hands you'll see. It has a unique fields on I D, which is email so we can use. That's to select the elements and then we can and sent text or characters to this elements so he can type automatically the email. So we open open the Firefox driver here, or you can use chrome and he wants No. Two euro. And then we want to get the elements, and the element is simply called email ideas email. So type driver those kinds elements by e mail, and we'll just call it email Fields has. You can use the method since keys to send characters, so that's well, rights your email. So if you to run these coats and just put a delay here, not the queen, before seeing opens and see, it starts to browser and then searches 40 fields with the idea that email. And then, since keys so starts typing in the text fuse, and I will just load that page and you'll see the email field here, and it will start typing automatically completely automatically. Right? So just takes a moment or some more to greatly noti what bitch? And there you go and see a type T email automatically, and we can do the same for the password fields. So if you click on the Arrow Select passport, you'll see the ideas possible worth. You can use that. So yes, copy these lines. Well, say possible it feels ends. The idea of the elements spots work right? Ideas possible? Er's Justin passwords. Now, if you run it again, you'll see it's completes Boat fields, sinister email field and the passport feels. And most of the Logan's only have these fields right. It is your email, went your passwords or user name and password, So this will allow you to look into most pages. See those fields right there, and the program will answer the imports. This takes a second or so depending on your browser. So in this case, the grocer's not that fast. See, both fields completed automatically, but we're still not locked in right, so we also need to click the log in button and you'll see the button right here. I see it. The idea off this field is submit button, so we'll use that to finds it button, and then we'll symphony elected button someone call the methods. Click and you'll see that if we run this program, it will throw a never because obviously the email and possibles are incorrect. But if you specify your email and your passwords, he will actually be locked into the webpage. Now you'll see we don't have the right email and passwords. Selenium, Giddens type death correctly. So that's because, obviously, these immigrant possible turn off correct. But either way to get elements you can simply use, he finds elementos idea our finds element by I d. If they have ideas, and then you can interact with the elements so you can send keys or click on them. If they don't have an idea, you can use the export. But in most cases, they will have an idea, a unique identifier to select the elements and then you can interact with them. 8. Screenshot: you can take a screenshot using the methods. Safe's underscores creations. Aspira Military specify A founding ends This foul will be stored in the same directory as your program, so in this case, it's well, lows. Start fire folks Loki, Webpage and then save Screenshots under the things Cree shows PNG on one thing to notice that this is only screenshots off visible window. And so, if you have a webpage, it will only take a screen shot for what's shown to the user inside the window. So the weapons could be much longer, but it only takes what's visible. And so it goes to a patient and see I took the world news space and people. It's quite alone webpage here, but it only takes a screenshot was visible so it takes a second or so ends, and it will save the special found. Now, once this is loading is completed, you'll see how was actually created. Now, after this are Syria. How to take a screen shots off the complete webpage show that includes everything that's not visible. Okay, we go now. It's safe discretion, But note how it doesn't include everything that summer webpage, right? So it stops right after what's shown to the user. And so to take a screenshot of the complete where pitch, who wants what these lines. So we simply gets the body off the webpage, and then we'll save it on a foul screenshot to come into this house. Clarity. And these lines, of course, will take the screen shots off the entire weapons so they think lose everything that's not visible in the browser window. And so you run the program. It will create a new foul called screenshot, too, which includes Screenshots off the entire pitch, so just loves for a second. Now ends. Once loaded, take discreet shows. No can take a moment to complete. Depending on your Internet connection. Open the file manager. You see another foul will pop up, which is much longer. Perhaps we can zoom in here, so we've got one foul. She's just the top off the weapons of off the faults, and we've got one. That's to complete weapons. So if we open that one, it sees the complete screenshots the weapons. So everything that's visible and invisible on the webpage where S T safe screenshot methods only takes above the force that doesn't include 17. So probably usually you want to have these lines of code to take Screenshots off the complete weapons. 9. Scrolling: make silly name, Scroll down the page automatically. So if you have the script here, we simply open a webpage. We wait two seconds and quiz. If you run this program, you see it will start a browser open. The webpage can be anywhere fish, and then it will quits. So some of these webpages so usually social media, Web pages and others they can have, like if you scroll to the bottom new constant with fear. And so it's just like to eat a roast. He kind of social network and several other science are doing this as well, and so, using this line, you can scroll to the bottom off my pace. So basically, we say to the browser executes scripts, which just JavaScript, and we can insert JavaScript that's directly inserted into the ground. Shippen's with weapons. So who is not asleep? And you'll see that once it started focusing webpage, which scrolls to the bottom weights again. So now it will Starkey in Russia, see it opens. His social media webpage can be anywhere. Peachtree Islam, New conference appearing on the bottom. So now it will scroll to the bottom automatically, and you'll see a new Huntington's loading. So just to be sure that this camp this can continue for a while. So let's save me, do it 10 times ends if you run it Now, you see new conference will appear each time, and Cellini in will keep scrolling them. So this is very useful when scraping, because in many Web pages like this, not all content is immediately loads. Maybe just 10 articles, 20 articles or 20 tweets 30 tweets, but not when you scroll to the bottom. New ones appear. So sometimes you want to scroll to the bottom each time just to collect more data. So now you'll see a scroll to the bottom. Those new contents cross to the bottom and keeps repeating that because we put it inside a Ford. So you see, we get more and more data that's loaded from the weapons and suing him will just keep doing this for a while. So sometimes you might want to do that strolling automatically, and you can do it just with the line. Of course, this is the complete skull to do that 10. Navigating: the Selenium browser Firefox browser, in this case, opening Firefox or Chrome in your face. Another keeps a history, and so for every pitch you open, you can go back and forwards. So let's say you open several pages, whichever paid she would like, and you can return to the last pitch can go first page on. After that, we can go for examples after this. So then you'll see it starts moving between those pages using the history. So if he around this program, it starts loading the browser and will visit every off those pages and then navigate between each pitch. And it can be useful and in your application, depending on what you're trying to achieve, how to load every page we specified hands. It may take a while, depending on your browser. Then you can use the methods forwards and back to navigate between death. So that's what you are going to see. Using the back function to go to the previous page does that twice. Calls to back function twice will do that again once this is completed, and then you'll see it moves forwards. So we're back on the second pitch so you can just you see, he beckoned forward function to navigate between patients. And if you want, you can also get current euro, so to pay to our own. You see this line so we just prints driver that currents underscore euro, which will give you the patients on it, is basically how can every gates, which is brow, suggest using beckoned for what mess? It's in the case of the current year out its particular useful because even if you don't use the back and forward methods. But if you click on buttons everywhere on the page using the exports, clicking on elements in between forms and so on, then at some point you may want to know on which picture actually looking. So then you can use to make dislike, to know which page you are all 11. Table Scraping: you can scrape entire tables using selenium, so just open any webpage with a table. It's case I picked this table about any table will do. So Table almost has several roasts on. Every row has one or more cells, so in this case, there are about 28 rows end. Each of them have different cells, and you can fetch that all of this data from anywhere page from any table into fighting coat and so pipes and variables. And so we start by simply opening D. You went pitch, then we need to know the table, identify and so open your developer tools. My presently have 12 key and go to your point to your table and you'll see in this case, the table doesn't have a unique identify here. It only has a class name. So in this case will have to use the class name misidentify. So sometimes you'll find it's for tables. They don't have a unique I d. And then you have to use class, name or all the tricks, and so we use the same class name. So I used the experts and say the class name should be equal to this. So you'll see this Wikipedia sort of all Juburi and so on. Steve Close Name. So I would say Close name Is that so with this line Fetch D the table and the next thing we want to do is to get get all of the rows and for each road at all of the cells. So for every row. So this is an HTML or a webpage row, and this is it cell. And we simply printed text so might look a little bit strange if you're unfamiliar. Which HTML codes. Which webpage coat? But essentially, tables are using the table T R and TD texts Tiara Rose and Tedy ourselves ends. We simply fetch each of them and printing. And finally, you might want to closure grousing. So let's save to say, some delay hair. Let's say three seconds and we'll equity program. So now we get the entire table. Randi program. Take a second or so two loads. Natsuki Patri loads and it'll fetch all off these data into pipes. So it's exactly the same day that from this table now, if you want to, you can of course, safe each of these variables into unique names. So you called Adam to one every. If you want a war to an object anyway, you can get it on the table. They don't just reducing exports and then specifying the table, identify their hands, rows and columns.