Open source multiplatform visual based testing | Noemi Ferrera | #SeConfLondon


Hi, can you hear me? Good. First step, crossed, good things. So, my name is Noemi Ferrera. If you struggle to say it, you
can say Naomi – it’s the same name. I don’t have a Who Am I slide because I’m
a bit shy, but that’s enough about me. I want to know a little bit more about yourselves,
so the first question first, who is hungry? Great. You’re alive. I will try to make it fast. If I speak, fast, someone stop me. If there is anything at any time that is confusing,
let me know as well. So, I’m going to talk today about open source
multi-platform visual basic testing tool which is, yes, is a mouthful, but, some people
may think this is about comparing images – it is and it is not at the same time. That’s why I call it visual-based testing. It means on top of retrieving objects, you’ll
be able to retrieve images. So I know how deep I can go into the coding
bits, who from here has some knowledge about code? Perfect. So, a lot of people. Right, so don’t worry if you don’t. You can still follow it. I have a lot of demos along the way. There are shorter videos, and I do this because
I’m a bit of a control freak! Hopefully, everything will work out. So, what is Airtest Project. It is an open source cross-platform tool developed
by a company in China called Netease, particularly in the games department. They found testing games was really, really
hard, one of the hardest applications to test. And we needed a solution in order for the
games to be able to be automated as well. So, in that way, we can save manual cost,
and all of that. This solution doesn’t work only for games,
the cross platform, I talk about that a bit further, and just so you know, it’s divided
in two, like two projects, and Airtest Project, if you look it up online, the – Airtest IDE
is how people maybe don’t have so much knowledge on test, not on testing, on the codebase,
they can still do the testing. Also, it has another model called Focal which
is like the back-end, and it’s based on Python. If you know Python, you should be able to
do the same stuff, even cooler, but I think the IDE is really got a to get you started
as well. It works in this platform here. As I said before, games were very important
for Netease, and, by the way, I forgot to say this was launched with Google in the Games
Development Conference in 2018. So it is a young project. There are still some bugs here and there,
but they’re really fast in fixing them, and, actually, it works really well. I have to say. So, I’m very fond of this project. As I say, it works in Android and windows
and iOS. iOS, you can set it up in your Mac to do some Selenium tests, tests, but if you
want to use the visual options, then it would not work, so, for the visual options, it’s
better if you use the Windows or if you use the Android, or the iPhone. For games as well, if you’re wondering what
sort of platform to use for games, there are unity and – but it’s also something that you
can extend, and use your own kind of tools on top of that. So, let’s see it in action. So, I have to come here because I have to
click Play. To download the project, you have to do to
airtest.com. You have a big download button there. I’m doing here for Mac and this is on a … licence,
but, you can also download it for Windows. If you download it with Mac, you get a DMD,
if Windows, you get a zip file. This is easy to get started. We will get a little bit more in a no, I did
not minute. Next. There we go. So, let’s see how it works with Selenium,
because we are in a Selenium conference, so, how can it make it work with it? First of all, we need to configure the Chrome
app. You can also do Firefox, so, for Windows,
it’s under this path here: Programme Files, Google, Chrome. We click on the exe file. Not on the Chrome driver. That’s set and good to go. Under Mac, it’s under the Applications folder. So, let’s start with it. The first thing I’m doing here is a bit fast
but I’m creating a new file, and I’m going to call it Selenium Conference London 2019. I know I’m missing an “n” and I know I put
18. I don’t know what year it is! It is fine. I fixed one of the two things. It’s going a bit fast, so I will run it again
if you don’t mind. So the first thing is I click on the globe. I’m going to stop it and start again because
it will be fast. All right, so the first thing that I did,
is that I have under Windows I selection of devices, because the devices is the window
that we will use for Android and iOS. It takes a lot of screen, so I take it off. I took on the Selenium window which is the
one you see on the left-hand side that has the objects which you use to start the testing. And then, it’s when I create the new file. And then the next stem I’m going to do is
that I’m going to click on the globe that appears on the top left hand, and that’s going
to start the Chrome browser for me, and it is also going to allow me to add this code. That little message there is would you like
to add the code? You need to know the code by heart. It’s easy to get started. And then I just navigate to the web that I
want to start, and I click Start Web, and that will put the code for me to start the
web. And then I just, like, play around. I click a touch. It’s like a click. And, if you know I’ve been with, I think the
same name, they also use Touch, and in here, what I’m doing is instead of that, I used
a little computer that is in the middle way, and that will tell me all this information
that is here in the local viewer. That gives me information on the DOM of the
object. And in here, I’m just creating a couple of
sent keys method. And because I mean, what is the presentation
if it doesn’t have any cat pictures, right? I’m going to look for cats. And then, the next thing that I want to do
is use the – this bit is a bit important. So, I – I’m going to run it again so everyone
has seen it. So what I do is, instead of that, I’m going
to click on the Airtest Touch. This is the visual touch. Wait for it. Good things make themselves wait. Again, I’m creating the methods here to set
the keys. I’m using the same object. Usually, it tries to look for the ID. If it doesn’t find it, it looks for the path
and that is the code that I generate. Now, this is doing manual – just so I can
navigate to the next space because I know this is the next thing. I want to find this button there. So I use the Airtest Touch. I select the button that will click it and
I get the method there. As you can see, there’s an image on there
which is that now, it’s in the back, it’s using artificial intelligence to find the
image in the website. It’s very, very handy in the case that you
have an object, but you cannot find any element, or the element changes a lot, and you can’t
find it in the screen, so then you can get the screen shot and find it by screen shot. This for me is the winner part of this project. And, the last thing that I did is, like, added
the driver manually, so Chrome will close, and I just run it now, so this is all act. Right now, I’m not touching the mouse. It was a bit fast. But, just so you know as well, that, you know,
it can be – it can fail as well. First of all, I’m going to wait a little bit. I’m going to wait for the Chrome to – like,
I use the same because I’m half dyslexic sometimes so I can’t type. I just copy the one. It’s the same as if you create the sleep method. You can create the sleep method as well. Whatever you do in Python, you can still do
here. And so now, you should, when I run, you should
take a little bit longer, and I stay there. I should probably have turned off this bit. So, yes, this, the driver won’t close so fast,
and then you can see the image being selected. So, yes, that’s cool. Right, Selenium, it’s similar to Selenium. It has a couple of other things. There’s an IDE so you can click the object
that is cool. It has a visual component which means you
can select the objects by selecting, like taking a screen shot of it. Yes, but, you know, it’s why is it interesting? Well, this is interesting in the sense of
the multi-platform test. So, now I’m going to create another new file. And this time, I’m going to do a test that
has Selenium and windows in the same test. So you don’t need to open another test or
anything. With the same test, you can do stuff that
you did in Selenium, and you can do the stuff in Windows as well. In this case, and, as you can see now, I added
the device Windows, which is on the right-hand side, and I connected to a search for a windows,
and I second to my panes in the Windows machine. We’re going to automate panes. So, instead of going through all the tests,
which could be there, but I’m going to summarise a little bit, and I’m just going to go to
this website, and I’m going to download the image. So, again, I add the test for the Selenium. I open it automatically, I get the, I start
the web – I start the web on my browser, and I just now, I’m using the recording functionality. You also have record the test, so you don’t
need to go touch and whatever selecting and input, you can just do recording and do whatever
you need to do and you automatically would generate the code. In fairness, that one works flakily sometimes
so I usually prefer to add the patterns one by one. Before I sleep for five seconds, I quit the
driver, and that’s when my Windows test starts. I’m selecting now an object from Paint which
is the file. And in here, the next step will pop up a window,
so now in the IDE, you can’t really use Windows if you have a connected one, but in Python,
you can. You should be able to assess the child of
the window that you connected previously and you should do your screen shots with that. Because I can’t do it on the IDE and I wanted
to show how IDE works, I’m going to start doing key events. It’s a bit hackie, but it works. I’m just using this code here to add a couple
of those, and an intro. That will take me in the Windows with the
open file for me. After that, I need to clip tab nine times
– I counted – in order to get to the right image. I click enter once and that will take me into
the place. That is tabbing nine times, four times down
and enter, and that should take me into the right image. I’m going to add that. It is a bit hackie, but it works. If I were not to use the IDE, if I went to
the Python code, I could do this as well automatically. I also thought after you click File, you can
open by click O but I didn’t come up with that when I was doing the recordings. That’s mostly done. I’ll go a bit faster. I think you get the picture of this one. And the last one is an enter. Right, so, if we execute now this, we should
see our browser being open, and our Selenium test being executed. Just as in any Selenium test. After that, we should be able to see or actions
being executed on the window that we have selected which, in this case, is a paint where
you can select any windows. I don’t know, like, I’ve seen a very good
talk in Simon’s present – I’ve seen a good dog in Simon’s presentation, and now a cute
cat and no-one is going “aww”! This one is fading in this case because the
FindElement was, if you can see the bottom, I was clicking on the things at the bottom,
and it didn’t like it. This happens a lot in Selenium. If of you worked with Selenium, you’ve probably
seen this issue. I fixed it in this case by just getting the
entire diff, so the big button but I could have fixed it as well by taking a screen shot
fit which is very handy. The only problem with the screen shot, I have
to say, is that sometimes they fend a lot on the resolution. If you change your computer and you take the
same screen, it may fail. There’s a way of fixing it here. If you run it again, it opens the Chrome browser,
and even though I’m going a bit fast, this is really repetitive, so you’ll get the idea. And then you download it, and there you go,
you have it downloaded. It should go to the file. So, it quits the browser, and I did that so
you can see. Because this is very interesting, the IDE,
but it’s a bit limited when you have the smaller screen, so it’s better to use a couple of
screens that is much better. But still, you know, for showing purposes,
it’s like that. As you can see, it opened the image, and the
image is showing there. It’s a bit too big. But there we go. So the-the-cool things that you can do. This is for windows, I clicked there, which
is on the options side to correct the connect the window but not embed it inside, so my
window is still open as it was before. It doesn’t go inside the IDE. It is a bit handier. I get the smaller image so you can see. So, a couple of things that I can do with
Windows, and again, I can do this after the Selenium test, it’s no issue, but it’s cleaner,
I commented on the previous code. Here, I’m selecting the arrow to do some drawings. And, in here, I’m going to select the vertical
line that is on the click board, and, if you double-click on the image, you get the information
of the resolution, the position of where the image is being recognised. So, the red dot will show you exactly where
the image is being recognised so you can change the parameters and play around and see where
exactly the image has been recognised. So, for example, if I want to draw an arrow
a bit, I may be interested in having the position lower – higher, actually. If you want the colours back or white, I tend
to prefer black and white but some tests will necessarily change colour, so maybe you want
to have the two colours too defensive. When I press run, it draws the arrow. If I change the position, so the recognition
will be a bit higher, now the arrow should be drawn only a bit higher. And there we go. I’m a kitty cat! So, all right. So the demo. As we’ve seen before, you can develop the
IDE – you can download the IDE. You can have a lot of documentation in there. If you were to deep debug things, you could
clone the code from the Airtest project. And, if you want to do iOS, so you want to
connect it to your mobile device, you can use the iOS which is in the other GitHub URL,
and last but not least, if you want to do it in Android, you can also develop the iterated
Studio for that to work. For this presentation, I’m going to skip that
part, which I hope you’re happen about because it means we go to lunch earlier. And just so you know, that you can do it,
right? This is very convenient. All this stuff, you can also work a bit with
the Firefox, not only with Chrome, only that Chrome works that little bit better, especially
for recording. So now you’re probably thinking, right, this
seems very cool, but there must be a lot of issues with it. For example, when you create, like a screen
shot, you remove them, and your screen shot stays in your computer and you have your computer
full of useless screen shots, the tests might not be reliable when the UI changes, if you
have a different UI, you need to support two of them, then you need to be changing the
screen shot, copying and pasting the screen shot, creating another test which is exactly
the same with a different screen shot, this is where visual base testing doesn’t interest
a lot of people because this makes it hard to maintain. So how do you configure your set-up in order
to maintain something like that?? In the end, it’s a lot of manual process which
means why would you use? You’re probably better off doing it manually. And, in the last also the report, so how do
I get the information of what fails and why it fails? There is someone in the IDE, but if you automated
with Python, how do you do it? There are solutions for this which I think
are pretty cool cool as well. So, the first thing I want to show is the
Airtest project solution, so, when you create a new test, it creates a new folder which
is .ai – I was doing this in VMWare, actually, so I didn’t
get to install any better tools, you go but you git the idea. It’s a Python file and a bunch of screen shots. So how do I remove the convenient shots that
I’m not using? It is super easy, if you remove a screen shot
and you decide to remove it, it’s still in your folder. The screen shot has been created at the bottom. I right-click and delete the screen shot. That’s it. Your screen shots are gone. I think that that is very cool. I don’t see a lot of excitement, but maybe
I’m talking fast! This is the same stuff. So, what about the issue that I was telling
before about when I have two buttons and they look different? I’m doing this test with Duolingo. Do any of you study languages? Yes, it’s a bit, an app that tells you all
the time that you’re not studying! But, you know, it’s interesting for the test. So, in here, on the right-hand side, I have
Duolingo Windows app and I will show the web and the buttons for getting it started. It looks very, very different in both. In this case, I’m going going to add the devices
and the Selenium window, and I’m going to search window, refresh, and select the Duolingo
one. So now I’m ready for doing the Windows test
and I have the Windows for Selenium there, so I’m ready for the Selenium test, and I
create a new file again. So, it’s very repetitive, so you can see it
now. And, in here, I’m going to open the browser
as before, and I’m going to click yes so that is the code for me, and I’m going to navigate
to duolingo.com and start web, and that will take me to that web. I don’t even need to type it twice. Here, I’m going to create a touch which is
just getting the screen shot for the Selenium. And, as I say, it’s a bit flaky sometimes. Double-click, and there we go. And, I’m going to do the same for the Windows
application. I select the image that I want to be recognised. I double-click it. And is it going to work? Probably didn’t click fast enough, and there
we go. So, now I have the two touch, right, and they
look absolutely different. So, if I were to run it, it will click in
that too. If I right-click and click on the render code
or image, I can see instead of the image, I can see the code goes below, and that is
pointing to the image that is safe on my computer, it’s telling the resolution, and it’s staying
the position in this case, and that’s very powerful because it means that you can use
the same image, you can change the resolution, you can take another screen shot and use the
other screen shot as well, and this down here is my favourite part, probably of the whole
presentation, which is I’m actually creating variables to keep the screen shot, so I’m
keeping shows screen shots I’ve created in different variables which means I can use
the variables in the touch methods, which is, first, much cleaner, and, secondly, much
more efficient, because it means that if I have, like, if I were to create five different
screen shots, I can have them all in five different variables and use the one that I’m
interested in them. But also, it means that well, this is Python. In Python, I can do loops. So, I can create the loop. I can check each of the variables that I’m
going through. Just for you to know that it works, so, if
I click Run in this case, it will create the two variables. So you need to have your windows hooked up
for this to work, and then it will just, as before, it will open the browser file, and,
whenever it goes to the touch method, instead of having the image in there, it will go directly
to the variable, and we should see that it clicks on Get Started in both screens. Any time. There we go. And it will. So all done. As I said before, you can add this stuff into
loops which is what I did here. I commented out the previous methods, and
I created a for-loop with the two variables, so, if the image in the Windows exists, touch
it, and then we can access the loop. We can do the same with the Chrome as well. In this case, it’s had not collaborative. You can have five different Windows, images,
five different web images, and you can play around with this stuff. I think it’s pretty cool. So, in this case, because the image Windows
are first and my touch method is the Windows one, not the Chrome one, it will navigate
to Duolingo but now will click on the right-hand side which is on the Windows side. And there we go. So I didn’t click on the other one. I just clicked on the Windows. So, in order to make the test reliable, this
is a bunch of, it actually came from my blog, by the way. Visit my blog, please. I get a few templates of the images, so you
can create all the variables you want, and then you can have some methods, and if you
have already picked up the name, the name of this class’s page, maybe you know where
I’m going to. Who knows what is page object model? Cool. You’re still awake! So I’m going to present you the enhanced page
object model which is the page object model using screen shots. So as we do with our page object model, you’re
going to have the pages with the objects and the basis method in one side and you get the
functionality for your test in the other side. The functionality of the test, I’m going to
leave it on the air file and I’m going to create a new file for the pages. I’m also going to – you can separate it in
folders as well, and create your screen shots in the same folder as the page. Just to keep it cleaner. After how you want to develop, software, a
code that you may need which I’m preparing here which is in Python, how do you assess
a different file? So, this is a kind of thing that is not part
of the test ID. It is something that I worked on. So, again, it’s a bit hacky but we can go
outside and create the joint paths with the one in your page folder. And, in here, you can see how I can call the
page folder, and I just create the different variables, and I just can do a bit of code. So let’s see it working because it is probably
clearer this way. So, again, create a new file. In this case, I’m going to create a Python
file, and I’m going to do it inside the Air folder, in this case, to make it simple, and
it’s going to be two pages where I’m going to keep the pages of the Duolingo page. The Duolingo window, sorry. So, I take that away, and I’m going to – this
is automatic stuff that it creates with the null created Python file, but in this case,
I don’t want the Python file to be my main file so I can get rid of it. In this case, I’m going to copy directly from
the other file, and this these implementation. So, now again, the images you can go back
and forth between the images and the code, and I have a couple of click methods and the
stuff that I don’t really need because it’s already up there. It is important that I import the API, and
then the next bit, it is my test file. So my test file, I’m going to do what I did
before. So go in order to assess with the next folder
added to my path. And which is what I’m doing here. And it’s a bit of code that I showed you in
a couple of slides before. After that, I’m going to access, so create
my page and access to my page a file from this other file. So this is kind of what you have to do all
the time. Sometimes, I go too fast and sometimes it’s
too slow. I’m almost done. You can have at least five extra minutes at
least! Maybe I can go a bit faster here if you guys
want. So this is the part where I create the page. So I have to import the Duo page, and from
there, the class was called page so that is what I’m calling page, and this is just to
show that you can actually access the images directly because like they’re part of the
class created before, and you can also access the methods. So I’m going to go a bit faster because it’s
too slow now. So, we just click web, we pass the driver
in this case, which is something that we created before, so, it is exactly like any other page
object model only that you have your screen shots kept, and I need to get the Windows
connected, as I said before, because it will break, even though you’re not testing anything
with Windows, and you’re declaring it, it will break. Once that is happening, I just run it, and
now it should only click on the Duolingo because it’s the only method that I’m calling on the
web which is the left-hand side. So any time now. There we go. So, that way, you can create maintainable
and reliable codes because you can add your screen shots together, and you can separate
the functionality, you can get someone to create pages for you, and you can focus maybe
on the tests, whichever works for your organisation. So, another thing that I mentioned was the
reports, so you can get reports like proper reports, and be by clicking Control L or click
“run view report”. Now they look like a screen shot, they look
like – and I think, in this screen shot, you can see where the steps were going, where
the objects were clicked. If you have anything visual, you see exactly
which part of the screen was clicked. If you have any issues, this is where you
can troubleshoot them. So this is like, it’s a Chinese project, so
it will have … group which we don’t use so much, but I added on the bottom here, you
can also have a Google group which is what we are more supporting at the moment and there’s
also a Slack channel but that channel will end and we will keep the Google Group. Also, in case you miss it, this has been in
all my slides, this is my Twitter handle, and so you can contact me if you also have
any issues, and I will try to help you out. So, conclusion: so this is a new tool. I think it’s pretty cool. It’s not here because it’s being developed
in China, but it works. And it’s very convenient for any automation. It’s very cool the way that you can use it
with the recordings, so it is cool for people who are starting. It has an IDE for Selenium, so you can use
it for Selenium and other platforms. You can use it together with two platforms
in the same script. It has a multiple ways of accessing applications,
which is cool. It’s using Python in the background which
is one of the most loved languages nowadays, so, I mean, if you don’t know Python, it’s
very easy, so you should be fine. And it’s possible to design reliable and maintainable
tests using the enhanced page model, and which at least is possible. It’s not the easiest way to implement, but
still, once you set it up, it’s easier to keep using it. And, the cool thing is that you can keep using
the screen shot through the process as well. As I say, five minutes you have, unless you
want to ask me some questions.

1 thought on “Open source multiplatform visual based testing | Noemi Ferrera | #SeConfLondon

  1. Extraordinary Work, I really enjoyed it!, See this New Album 'Monish Jasbird – Death Blow', channel link www.youtube.com/channel/UCv_x5rlxirO-WKjLIyk6okQ?sub_confirmation=1 , you may like it 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *