Beckhoff#TwinCAT x RoboDK x Pi4 x Coral USB ML – Image Classification

Hello! In this Tutorial I will explain how to use RoboDK x TwinCAT x Raspberry Pi4 x Coral USB to implement a Machine Learning Application for image Classification.

Let me Introduce myself again. I am an Engineer that comes from Hong Kong and now working in Japan.I do not have a University education Background, and this is also a big reason for me to try harder to learn every day.

After reading this post , I hope you can find some “idea” to integrate the edge device in your Machine and using the edge device is not as hard as we think.

To finish this task, there are so many small goals that I splitted:

  • Setup the RoboDK Station
  • Setup the Raspberry and Edge Device
  • The Handshake between RoboDK and TwinCAT
  • The Handshake between TwinCAT and Raspberry
  • Image Transfer from TwinCAT to Raspberry
  • Visualization by using TF1800 in TwinCAT
  • Know what is Transfer-Learning
  • Setup the OPC UA Communication between device
    • build an OPC UA Server by TF6100 in TwinCAT
    • build an OPC UA Client in RobDK and Raspberry PI
    • Read/Write the structure Node

It is very different to write all these topics from 0 to 100, I will provide the post link that I wrote before and please reference it to finish some small task.

Sure you can download the source-code at the end of the post.

Transfer-learning?

This post had a very clear explanation of What transfer-learning is.

Transfer learning is a machine learning method where a model developed for a task is reused as the starting point for a model on a second task.

It is a popular approach in deep learning where pre-trained models are used as the starting point on computer vision and natural language processing tasks given the vast compute and time resources required to develop neural network models on these problems and from the huge jumps in skill that they provide on related problems.

https://machinelearningmastery.com/transfer-learning-for-deep-learning/

In this Tutorial I will use the model from Coral that supports classification of 1000 objects.


Configuration

Here is the Configuration of this tutorial. We will configure the TwinCAT as an OPCUA Server by TF6100, And configure the RoboDK and Raspberry as OPC UA Client , and access the Node inside the Server.

While the virtual robot finishes the image snap, TwinCAT will send the images to Raspberry pi with FTP protocols.TwinCAT is FTP Client by using TF6300 and Raspberry is a FTP Server.

Edge Device,Coral USB is plugged into Raspberry pi in USB3.0 to speed up the Image Classification.Finally , the classification results are sent back to TwinCAT with OPCUA, and Display by TF1800.

In this example, TwinCAT and RoboDk are installed on the same PC.

Operation Concept

In this project, we have Manual Mode and Auto Mode.

Manual Mode

Here is the Flow of Manual Mode.The operator can select which image that would like to classification, then press the send button to send to the raspberry.

After the image is received and classified , the result will send back to TwinCAT .

Auto Mode

Auto Mode is very Similar to Manual Mode but just has the handshake part.

TwinCAT will wait for the Program end flag from RoboDK.

If Program End Flag from RoboDK=1, TwinCAT will search all the images that included with name “Robo_*.jpg” and list up.Then TwincAT will transfer the image to raspberry one by one to get the classification result.

TwinCAT Side

There are 3 configurations that you need to set up here. They are TF6300 FTP Client, TF6100 OPC UA Server and the TF1800 Screen.

FTP Client Setup

Please reference the following Link:

Beckhoff#Using TwinCAT TF6300 to Upload file to Raspberry

OPCUA Server Setup

Please reference the following Link:

Beckhoff#Using TwinCAT TF6300 to Upload file to Raspberry

Allow to access With Structure

Insert this attribute to all the OPCUA Nodes.

{attribute ‘OPC.UA.DA.StructuredType’ := ‘1’}

Visualization

Please reference the following Link, you will know how to search files in the Directory and configure the screen.

Beckhoff#Using FB_EnumFindFileList to Find Files inside Directory

Manual Screen

Auto Screen

Code

DUT

DUT_OPCUA_PI4_ClassificationResult

The Data type for the Image Classification Result

TYPE DUT_OPCUA_PI4_ClassificationResult :
STRUCT
Label:STRING;
Score:REAL;
END_STRUCT
END_TYPE

DUT_OPCUA_PI4_TPUConfigurations

It is not being used now but planning for the Classification parameters.

TYPE DUT_OPCUA_PI4_TPUConfigurations :
STRUCT
top_k:UINT:=2;//Max number of classification results
Count:UINT:=5;//Number of times to run inferences
mean:REAL:=128.0;
std:REAL:=128.0;
threshold:REAL:=0.0;
END_STRUCT
END_TYPE

DUT_OPCUA_ReadFromPI4

The OPC UA Node for Raspberry pi4 to write the data.

TYPE DUT_OPCUA_ReadFromPI4 :
STRUCT

bClientReady:BOOL;
bTPUReady:BOOL;
bBusy:BOOL;
bFinished:BOOL;

arrResult:ARRAY[0..5]OF DUT_OPCUA_PI4_ClassificationResult;
END_STRUCT
END_TYPE

DUT_OPCUA_Write2PI4

The OPC UA Node to send the data to Raspberry PI.

TYPE DUT_OPCUA_Write2PI4 :
STRUCT
bStart2Classification:BOOL;
sImageName :STRING;
END_STRUCT
END_TYPE

DUT_OPCUA_ReadFromRoboDK

The OPC UA Node that is sent to RoboDK.

TYPE DUT_OPCUA_ReadFromRoboDK :
STRUCT
sImagePath:STRING;
bImageisSaved:BOOL;
bBusy:BOOL;
bEnd:BOOL;
END_STRUCT
END_TYPE

DUT_OPCUA_Write2RoboDK

The OPC UA Node that is received from RoboDK.

TYPE DUT_OPCUA_Write2RoboDK :
STRUCT
bStartProgram:BOOL;
END_STRUCT
END_TYPE

DUT_HMIFileEntry_Display

This is the DUT for HMI Display and Structured screen is used in this Project.

And also the T_FILETYPE is not easy to display in HMI, DT type is used.

TYPE DUT_HMIFileEntry_Display :
STRUCT
LastAccesTime,LastModificationTime,CreationTime:DT;
FileEntry:ST_FindFileEntry;
ImagePath:STRING;
END_STRUCT
END_TYPE

eFBStatus

The Enum status for Function Block status.

{attribute ‘qualified_only’}
{attribute ‘strict’}
TYPE eFBStatus :
(
Idle:=0
,Busy:=1
,Error:=2
);
END_TYPE

GVL

GVL

The variables for HMI Display and signal Exchange between different POU.

{attribute ‘qualified_only’}
VAR_GLOBAL
currentPath:STRING;
arrResult:ARRAY[0..49,0..5]OF DUT_OPCUA_PI4_ClassificationResult;
FindList:ARRAY[0..49]OF ST_FindFileEntry;
nFindFiles: UDINT;
bUpload:BOOL;
OutputIndex:INT;
ImagePath:STRING;
AutoRun:BOOL;
bTrigger:BOOL;
hmiAutoDisplay:ARRAY[0..5]OF DUT_OPCUA_PI4_ClassificationResult;
bReset:BOOL;
Processing:BOOL;
Processing2:BOOL;
Hide:BOOL;
END_VAR

GVL_OPCUAServer

The OPC Server Node.

{attribute ‘qualified_only’}
VAR_GLOBAL
{attribute ‘OPC.UA.DA’ := ‘1’}
{attribute ‘OPC.UA.DA.StructuredType’ := ‘1’}
ReadFromPI4:DUT_OPCUA_ReadFromPI4;
{attribute ‘OPC.UA.DA’ := ‘1’}
{attribute ‘OPC.UA.DA.StructuredType’ := ‘1’}
WriteToPI4:DUT_OPCUA_Write2PI4;
{attribute ‘OPC.UA.DA’ := ‘1’}
{attribute ‘OPC.UA.DA.StructuredType’ := ‘1’}
ReadFromRoboDK:DUT_OPCUA_ReadFromRoboDK;
{attribute ‘OPC.UA.DA’ := ‘1’}
{attribute ‘OPC.UA.DA.StructuredType’ := ‘1’}
Write2RoboDK:DUT_OPCUA_Write2RoboDK;
END_VAR

FB_FTPClient

Please reference the following Link:

Beckhoff#Using TwinCAT TF6300 to Upload file to Raspberry

pFileOpeartions

Please reference the following Link:

Beckhoff#Using FB_EnumFindFileList to Find Files inside Directory

VAR

VAR
EnumFindFileList:FB_EnumFindFileList;
FindList:ARRAY[0..49]OF ST_FindFileEntry;
sSearchPath:STRING;
bExecute:BOOL;
commnd:E_EnumCmdType;
bBusy:BOOL;
hmiIndex:INT:=0;
bError: BOOL;
nErrID: UDINT;
bEOE: BOOL;
nFindFiles: UDINT;
hmiDisplayFileEntry:DUT_HMIFileEntry_Display;
FileTime:DT;
sBasicPath:STRING:=’file:\\\C:\FTPFiles\’;
bDisplay:BOOL;
hmiIndexDisplayBackup:INT;
R_TRIG:R_TRIG;
AutoRun: BOOL;
F_TRIG:F_TRIG;
END_VAR


Program

AutoRun:=GVL.AutoRun;
IF NOT AutoRun THEN
sSearchPath:=’C:\FTPFiles\*.jpg’;
ELSE
sSearchPath:=’C:\FTPFiles\RoboDK*.jpg’;
END_IF

R_TRIG(CLK:=GVL_OPCUAServer.ReadFromRoboDK.bEnd);
IF R_TRIG.Q THEN
bExecute:=TRUE;
END_IF

EnumFindFileList(
sNetId:=”
,sPathName:=sSearchPath
,eCmd:=commnd
,pFindList:=ADR(FindList)
,cbFindList:=SIZEOF(FindList)
,bExecute:=bExecute OR R_TRIG.Q
,bBusy=>bBusy
,bError=>bError
,nErrID=>nErrID
,bEOE=>bEOE
,nFindFiles=>nFindFiles
);

IF EnumFindFileList.bExecute AND (EnumFindFileList.bEOE OR EnumFindFileList.bError)THEN
bExecute:=FALSE;
IF AutoRun  THEN
GVL.bTrigger:=TRUE;
END_IF
END_IF
IF NOT AutoRun THEN
IF hmiIndex >=0 AND
hmiIndex <=15 THEN
hmiDisplayFileEntry.FileEntry:=FindList[hmiIndex];
END_IF
IF hmiIndex <0 THEN
hmiIndex:=15;
ELSIF hmiIndex >15 OR hmiIndex > EnumFindFileList.nFindFiles-1  THEN
hmiIndex:=0;
END_IF

END_IF;



IF AutoRun THEN
IF hmiIndex >5 THEN
hmiIndex:=0;
END_IF
IF hmiIndex <0 THEN
hmiIndex:=5;
END_IF
hmiDisplayFileEntry.FileEntry:=GVL.FindList[hmiIndex];
GVL.hmiAutoDisplay[0]:=GVL.arrResult[hmiIndex,0];
GVL.hmiAutoDisplay[1]:=GVL.arrResult[hmiIndex,1];
END_IF

F_TRIG(CLK:=GVL.Processing);
IF F_TRIG.Q THEN
hmiIndex:=99;
END_IF
IF hmiIndex <>hmiIndexDisplayBackup THEN
hmiIndexDisplayBackup:=hmiIndex;
bDisplay:=TRUE;
END_IF

hmiDisplayFileEntry.CreationTime:=
FILETIME_TO_DT(
fileTime:= hmiDisplayFileEntry.FileEntry.creationTime);

hmiDisplayFileEntry.LastAccesTime:=
FILETIME_TO_DT(
fileTime:= hmiDisplayFileEntry.FileEntry.lastAccessTime);

hmiDisplayFileEntry.LastModificationTime:=
FILETIME_TO_DT(
fileTime:= hmiDisplayFileEntry.FileEntry.lastWriteTime);

hmiDisplayFileEntry.ImagePath:=CONCAT(STR1:=sBasicPath,STR2:=hmiDisplayFileEntry.FileEntry.sFileName);

GVL.currentPath:=hmiDisplayFileEntry.FileEntry.sFileName;
GVL.FindList:=FindList;
GVL.nFindFiles:=nFindFiles;


piHandShakeCheck

The Handshake between RoboDK and Raspberry and TwinCAT.

VAR

VAR
Count:INT;
R_TRIG:R_TRIG;
TON:TON;
iStep:INT;
i,j:INT;
bTrigger:BOOL;
OutIndex:INT;
testString:STRING;
bRun:BOOL;
bClear:BOOL;
END_VAR

Program

IF GVL_OPCUAServer.Write2RoboDK.bStartProgram 
AND GVL_OPCUAServer.ReadFromRoboDK.bBusy
AND GVL.AutoRun
THEN
GVL_OPCUAServer.Write2RoboDK.bStartProgram:=FALSE;
END_IF

IF NOT GVL.AutoRun THEN
GVL_OPCUAServer.Write2RoboDK.bStartProgram :=FALSE;
END_IF

IF GVL.bReset THEN
iStep:=0;
GVL.bTrigger:=FALSE;
GVL.bUpload:=FALSE;
GVL.bReset:=FALSE;
bRun:=FALSE;
OutIndex:=0;
END_IF

IF GVL.nFindFiles >0 AND GVL.bTrigger AND NOT bRun THEN
iStep:=1;
bRun:=TRUE;
GVL.bTrigger:=FALSE;
FOR i:=0 TO 5 DO
FOR j:=0 TO 5 DO
GVL.arrResult[i,j].Label:=”;
GVL.arrResult[i,j].Score:=0.0;
END_FOR
END_FOR
END_IF;

IF bClear THEN
FOR i:=0 TO 49 DO
FOR j:=0 TO 5 DO
GVL.arrResult[i,j].Label:=”;
GVL.arrResult[i,j].Score:=0.0;
END_FOR
END_FOR
bClear:=FALSE;
END_IF



CASE iStep OF
1:
GVL.OutputIndex:=OutIndex;
GVL.ImagePath:=GVL.FindList[OutIndex].sFileName;
GVL.bUpload:=TRUE;
IF GVL_OPCUAServer.ReadFromPI4.bBusy THEN
iStep:=2;
END_IF
2:
TON(in:=NOT GVL_OPCUAServer.ReadFromPI4.bBusy ,PT:=T#0.3S);
IF TON.Q THEN
TON(IN:=FALSE);
iStep:=3;
END_IF
GVL.bUpload:=FALSE;
3:
GVL.arrResult[OutIndex,0]:=GVL_OPCUAServer.ReadFromPI4.arrResult[0];
GVL.arrResult[OutIndex,1]:=GVL_OPCUAServer.ReadFromPI4.arrResult[1];
OutIndex:=OutIndex+1;
iStep:=99;
99:

IF OutIndex > GVL.nFindFiles-1 THEN
OutIndex:=0;
bRun:=FALSE;
iStep:=0;

ELSE
iStep:=1;
END_IF

END_CASE

GVL.Processing:=iStep=0 ;
GVL.Processing2:=NOT GVL_OPCUAServer.ReadFromRoboDK.bBusy;
GVL.Hide:=NOT GVL.Processing OR GVL_OPCUAServer.ReadFromRoboDK.bBusy;


Main

The program for FTP Control and send the command to Raspberry.

VAR

VAR
FTPBasicClientEx:FB_FTPClient;
bUpload:BOOL;
bDownload:BOOL;
iStep:INT;
hrUpload:DINT;
hrDownload:DINT;
TON:TON;
iFtpStep:INT;
F_TRIG_Upload,F_TRIG_Download:F_TRIG;
sSendImage:STRING;
sBasicSrcPath:STRING:=’C:\FTPFiles\’;
sBasicDestPath:STRING:=’/files/’;
Count:INT;
R_TRIG:R_TRIG;
AutoRun:BOOL;



END_VAR

Program

AutoRun:=GVL.AutoRun;
//FTP
//Upload Progress4
IF NOT AutoRun THEN
FTPBasicClientEx.SrcFilePath:=CONCAT(STR1:=sBasicSrcPath,STR2:=GVL.currentPath);
FTPBasicClientEx.DestFilePath:=CONCAT(STR1:=sBasicDestPath,STR2:=GVL.currentPath);
ELSE
FTPBasicClientEx.SrcFilePath:=CONCAT(STR1:=sBasicSrcPath,STR2:=GVL.ImagePath);
FTPBasicClientEx.DestFilePath:=CONCAT(STR1:=sBasicDestPath,STR2:=GVL.ImagePath);
END_IF
IF AutoRun THEN
bUpload:=GVL.bUpload;
END_IF
IF bUpload AND NOT bDownload AND iFtpStep=0 THEN
FTPBasicClientEx.UserName:=’pi’;
FTPBasicClientEx.Password:=’raspberry’;
FTPBasicClientEx.Host:=’192.168.3.14′;
FTPBasicClientEx.Port:=DEFAULT_FTP_PORT;
iFtpStep:=1;

END_IF
//Download Progress
IF bDownload AND NOT bUpload AND iFtpStep=0 THEN
FTPBasicClientEx.UserName:=’pi’;
FTPBasicClientEx.Password:=’raspberry’;
FTPBasicClientEx.Host:=’192.168.3.14′;
FTPBasicClientEx.Port:=DEFAULT_FTP_PORT;
FTPBasicClientEx.SrcFilePath:=’/files/log.txt’;
FTPBasicClientEx.DestFilePath:=’ C:\FTPFiles\log.txt’;
hrDownload:=FTPBasicClientEx.Download(bDownload);
iFtpStep:=2;
END_IF

//Function Block
FTPBasicClientEx();

//Status Control
CASE iFtpStep OF
1:
hrUpload:=FTPBasicClientEx.Upload(bUpload);

F_TRIG_Upload(CLK:= hrUpload = eFBStatus.Busy);
IF F_TRIG_Upload.Q THEN
iFtpStep:=0;
GVL_OPCUAServer.WriteToPI4.bStart2Classification:=TRUE;
GVL_OPCUAServer.WriteToPI4.sImageName:=FTPBasicClientEx.DestFilePath;
bUpload:=FALSE;
hrUpload:=FTPBasicClientEx.Upload(FALSE);
END_IF
2:
hrUpload:=FTPBasicClientEx.Download(bDownload);
F_TRIG_Download(CLK:= hrDownload = eFBStatus.Busy);

IF F_TRIG_Download.Q THEN
iFtpStep:=0;
hrUpload:=FTPBasicClientEx.Download(FALSE);
END_IF

END_CASE

//Flag Reset
IF bDownload AND hrDownload >0 THEN
bDownload:=FALSE;
END_IF
TON(
IN:=GVL_OPCUAServer.WriteToPI4.bStart2Classification
,PT:=T#0.4S
);
IF TON.Q THEN
GVL_OPCUAServer.WriteToPI4.bStart2Classification:=FALSE;
END_IF


//
pFileOpeartions();
piHandShakeCheck();


Raspberry PI4 Side

Please reference the following Link to easily configure your Raspberry. 

RaspberryPI#Enable the SSH Interface without start the OS
RaspberryPI#Set the static ip without starting the OS
Raspberrypi#Using VNC Viewer


FTP Sever Setup

Please reference the following Link:

Beckhoff#TwinCAT TF6300 でFTPServerにFileをUploadしよう


Coral USB Setup

Please reference the following Link:

Beckhoff#Using FB_EnumFindFileList to Find Files inside Directory

Model

This model is used in this time to classificate 1000 objects.

https://coral.ai/models/image-classification/


OPCUA Client Setup

We will use python-opcua in this tutorial.

https://github.com/FreeOpcUa/python-opcua

Install

pip install opcua

File Configuration


Code

import numpy as np
import time
from PIL import Image
from pycoral.adapters import classify
from pycoral.adapters import common
from pycoral.utils.dataset import read_label_file
from pycoral.utils.edgetpu import make_interpreter


from opcua import Client
from opcua import ua
import os.path
import datetime

#OPCUA Setup
ENDPOINT=”opc.tcp://192.168.3.124:4840″
client=Client(ENDPOINT)
NodeID2Server=’ns=4;s=GVL_OPCUAServer.ReadFromPI4′
NodeIDFromServer=’ns=4;s=GVL_OPCUAServer.WriteToPI4′
BASEPATH=’/home/pi/ftp’

#TPU Setup
LABELPATH=’imagenet_labels.txt’
MODELPATH=’mobilenet_v1_0.25_128_quant_edgetpu.tflite’
top_k=2
threshold=0.0
count=2
input_mean=128
input_std=128

DEBUG=False

def getCurrentTime():
    return str(datetime.datetime.now())

def writetext2File(message):
    with open(‘/home/pi/ftp/files/log.txt’, ‘a’) as log:
        msg=getCurrentTime()+’:’+message+’\n’
        print(msg)
        log.write(msg)

def main():
    try:
        client=Client(ENDPOINT)
        writetext2File(‘Connecting to OPCUA Server in EndPoint:’+ENDPOINT)
        client.connect()
        writetext2File(‘Connecting to OPCUA Server in EndPoint:’+ENDPOINT+’..OK!’)
        Node2Server=client.get_node(NodeID2Server)
        NodeFromServer=client.get_node(NodeIDFromServer)
        client.load_type_definitions()
       
        Node2ServerValue=Node2Server.get_value()
        NodeFromServerValue=NodeFromServer.get_value()
        Node2ServerValue.bTPUReady=False
        ReadyCount=0
        CommandReceived=False
       
        Step=0
       
        #TPU Init
        #Load the Label
        labels = read_label_file(LABELPATH) if LABELPATH else {}
        writetext2File(‘Label is Loaded.’+LABELPATH)
       
        #Load the Model
        interpreter = make_interpreter(*MODELPATH.split(‘@’))
        interpreter.allocate_tensors()
        writetext2File(‘Model is Loaded.’+MODELPATH)
       
        # Model must be uint8 quantized
        if common.input_details(interpreter, ‘dtype’) != np.uint8:
                raise ValueError(‘Only support uint8 input type.’)
       
        ImageFiles=None
       
        while True:
           
                NodeFromServerValue=NodeFromServer.get_value()
               
                ReadyCount=ReadyCount+1
                Node2ServerValue.bTPUReady=True
                Node2ServerValue.bClientReady=True
                if ReadyCount<=200:
                    Node2ServerValue.bClientReady=False
                else:
                    Node2ServerValue.bClientReady=True
                    ReadyCount=ReadyCount+1
                    if ReadyCount >=400:
                        ReadyCount=0
                       
               
                if NodeFromServerValue.bStart2Classification and not CommandReceived:
                    #Command from OPCUA Server
                    writetext2File(‘Command is recevied..’)
                    print(‘Command is received..’)
                    CommandReceived=True
                   
                    writetext2File(‘Path is :’+BASEPATH+NodeFromServerValue.sImageName)
                    print(“path with:”+BASEPATH+NodeFromServerValue.sImageName)
                   
                    #Node2Server.set_value(ua.DataValue(Node2ServerValue))
                   
                    ImageFiles=os.path.isfile(BASEPATH+NodeFromServerValue.sImageName)
                    print(‘—-‘)
                   

                   
                if CommandReceived and ImageFiles :
                    Node2ServerValue.bBusy=True
                    #Node2Server.set_value(ua.DataValue(Node2ServerValue))
                    print(‘Image is found..’)
                    writetext2File(‘Path is found.’)
                    if Step == 2:
                            i=0
                            # Run inference
                            writetext2File(‘Inferenice with time.,The first inference on Edge TPU is slow because it includes loading the model into Edge TPU memory.’)
                            for _ in range(count):
                                    start = time.perf_counter()
                                    interpreter.invoke()
                                    inference_time = time.perf_counter() – start
                                    classes = classify.get_classes(interpreter, top_k, threshold)
                                    writetext2File(‘%.1fms’ % (inference_time * 1000))
                            for c in classes:
                     
                                    writetext2File(‘%s: %.5f’ % (labels.get(c.id, c.id), c.score))
                                    Node2ServerValue.arrResult[i].Label=labels.get(c.id, c.id)
                                    Node2ServerValue.arrResult[i].Score=c.score
                                    i=i+1
                            Step=99
                    if Step == 1:
                            params = common.input_details(interpreter, ‘quantization_parameters’)
                            scale = params[‘scales’]
                            zero_point = params[‘zero_points’]
                            mean = input_mean
                            std = input_std
                            writetext2File(‘Step is 1,Initing the Parameters.’)
                            if abs(scale * std – 1) < 1e-5 and abs(mean – zero_point) < 1e-5:
                                    # Input data does not require preprocessing.
                                    common.set_input(interpreter, image)
                            else:
                                    # Input data requires preprocessing
                                    normalized_input = (np.asarray(image) – mean) / (std * scale) + zero_point
                                    np.clip(normalized_input, 0, 255, out=normalized_input)
                                    common.set_input(interpreter, normalized_input.astype(np.uint8))
                            Step=2
                   
                    if Step == 0:
                            size = common.input_size(interpreter)
                            s=BASEPATH+NodeFromServerValue.sImageName
                            image = Image.open(s).convert(‘RGB’).resize(size, Image.ANTIALIAS)
                            Step=1
                            writetext2File(‘Step is 0,Got the image from TwinCAT and Open it.’)
                            for r in Node2ServerValue.arrResult:
                                    r.Label=”
                                    r.Score=0.0

                #not NodeFromServerValue.bStart2Classification and
                if Step > 0 and Step <99:
                    Node2ServerValue.bBusy=True
                    print(‘–t’)
                else:
                    CommandReceived=False
                    Node2ServerValue.bBusy=False
                    Step=0
                    ImageFiles=None      

                   
                Node2Server.set_value(ua.DataValue(Node2ServerValue))
                   


    finally:
        client.disconnect()
        writetext2File(‘Disconnected from OPCUA Sever.’)

if __name__ == ‘__main__’:
  writetext2File(‘Script is started.’)
  main()


RoboDK

The role of RoboDk in this project is only while in Auto Mode.RoboDK is not necessary in here but thanks for their user-friendly interface,python api and 2D Camera Simulation interface, I will test more operation between Robot ,TwinCAT,Camera and be closely to the real environment.


StationSetup

The station is operated to take the box and move to the couples Snap shot position and snap it , saved to the file system.

2D Camera Simulation Interface

Please reference the following Link:

RoboDK#Using 2D Camera Simulation Interface

OPCUA Client Setup

I am using the same library as Raspberry.

https://github.com/FreeOpcUa/python-opcua

Please reference the following Link:

RoboDK#Start Program from TwinCAT By OPCUA

Install

pip install opcua


Code

from robodk import robolink    # RoboDK API
from robodk import robomath    # Robot toolbox
import datetime
import time
from opcua import Client,ua
import random

RDK = robolink.Robolink()


UR=RDK.Item(“UR5”)
CameraPosition1=RDK.Item(‘tCamera1’)
CameraPosition2=RDK.Item(‘tCamera2’)
CameraPosition3=RDK.Item(‘tCamera3’)
CameraPosition4=RDK.Item(‘tCamera4’)
CameraPosition5=RDK.Item(‘tCamera5’)
CameraPosition6=RDK.Item(‘tCamera6’)
Prog=RDK.Item(‘pMain’)
ENDPOINT=’opc.tcp://127.0.0.1:4840′
NODEID2Server=’ns=4;s=GVL_OPCUAServer.ReadFromRoboDK’
NODEIDFromServer=’ns=4;s=GVL_OPCUAServer.Write2RoboDK’

client=Client(ENDPOINT)
client.connect()

cam_item = RDK.Item(“Camera1”)
if not cam_item.Valid():
    print(‘No camera..’)
    quit()

snapshot_file=’C:\FTPFiles\p1.jpg’
#Make Dir
#ret=RDK.Cam2D_Snapshot(snapshot_file,cam_item)
         
def createImagePath(s):
    return “C:\FTPFiles\RoboDK”+getCurrentTime()+”.jpg”

def getCurrentTime():
    return time.strftime(‘%Y_%m_%d_%H_%M_%S_’+str(random.randint(0,99)))


def Sanp(Path,Cam):
    return RDK.Cam2D_Snapshot(Path,Cam)

flag=[False,False,False,False,False,False,]

client.load_type_definitions()
Node2Server=client.get_node(NODEID2Server)

NodeFromServer=client.get_node(NODEIDFromServer)

NodeFromServerValue=NodeFromServer.get_value()
Node2ServerValue=Node2Server.get_value()
counts=0
start=0
end=0
_busy=False
while True:
    NodeFromServerValue=NodeFromServer.get_value()
    Node2ServerValue=Node2Server.get_value()
    Pose=UR.Pose()
    if not _busy and NodeFromServerValue.bStartProgram :
        _busy=True
        value = Prog.RunProgram()
    if Prog.Busy():
       
        Node2ServerValue.bBusy=True
        if Pose == CameraPosition1.Pose() and not flag[0]:
            ret=Sanp(createImagePath(“001”),cam_item)
            if ret :
                UR.setDO(‘CamIO_1’,1)
                flag[0]=True
        elif Pose == CameraPosition2.Pose()and not flag[1]:
            ret=Sanp(createImagePath(“002”),cam_item)
            if ret:
                UR.setDO(‘CamIO_2’,1)
                flag[1]=True
        elif Pose == CameraPosition3.Pose()and not flag[2]:
            ret=Sanp(createImagePath(“003”),cam_item)
            if ret:
                UR.setDO(‘CamIO_3’,1)
                flag[2]=True
        elif Pose == CameraPosition4.Pose()and not flag[3]:
            ret=Sanp(createImagePath(“004”),cam_item)
            if ret:
                UR.setDO(‘CamIO_4’,1)
                flag[3]=True
        elif Pose == CameraPosition5.Pose()and not flag[4]:
            ret=Sanp(createImagePath(“005”),cam_item)
            if ret:
                UR.setDO(‘CamIO_5’,1)
                flag[4]=True
        elif Pose == CameraPosition6.Pose()and not flag[5]:
            ret=Sanp(createImagePath(“006”),cam_item)
            if ret:
                UR.setDO(‘CamIO_6’,1)
                flag[5]=True
        if flag[0] and flag[1] and flag[2] and flag[3] and flag[4] and flag[5]:
            Node2ServerValue.bEnd=True
            start=time.time()
    else:
        Node2ServerValue.bBusy=False
        for i in range(0,6):
            flag[i]=False
    if Node2ServerValue.bEnd:
        if time.time()-start > 2:
            Node2ServerValue.bEnd=False
            _busy=False
    Node2Server.set_value(ua.DataValue(Node2ServerValue))

Result-Manual

Result-Auto

Source Code

https://github.com/soup01Threes/TwinCAT3/blob/main/DemoProject_TwinCATxRoboDKxRaspberryxML.zip

Finally

In Conclusion, There is an issue that I am facing now and still finding the solutions.

Image Transfer from TwinCAT to Raspberry time is too long.If there is some other method to share files between PC and Raspberry, would you please inform me?

My first plan is using TwinCAT TF3800(Machine Learning Function).Although the conversion between tfile to onnx is successful, I can not convert this onnx model to Beckhoff XML format by using these tools. The error shows something like “Format not supported”(actually I forgot it..).After couple of times of try and error, I choose the configuration in this tutorial – using Edge Devices.

If the conversion between onnx to Beckhoff XML format, the next challenges would be – How to pass the 128x128x images to the TwinCAT and integrated with Tc3_Vision.

And Also – It may be TF3800 does not support Tensorflow Lite currently.

The reason that I am using TwinCAT – They  support so many different drivers without additional hardware.we change to ModbusTCP/EIP or any other protocols depending on the applications.

When I was trying to use the Coral TPU in my Windows PC with TwinCAT Runtime, the installation could not be finished because of some errors. But I can install it with my Raspberry Pi4 in 5 mins without any problem.That is the reason why I use raspberry pi.

In this Post Coral USB is used,But you can also choose other edge devices:)

RobDK is installed in the PC with TwinCAT, but you can also install it in the RaspberryPI.

But for the image processing reason, integrating into RoboDK may be better.You do not need to use FTP to send the image anymore.

Also RoboDK is not a must have option here but Thanks for their user-friendly interface and python api, I can implement my station in just 10 mins.

The important thing that i know while i played with this project – you need to split the big goal to so many small goal and clear it one by one.While you make sandwich,you will wash and cut the vegetables, cut the hams and bread , make the egg salad and then finally you will combine it  to Sandwich.

And Also, So many people(me) although they know how to wash and cut the vegetables, cut the hams and bread, It is very different to have a mind breaking – Hey! I can combine these to make a sandwich!!  – connect all the dots that you have now, and make a new picture.

Finally, I am using the model to make the image classification.  There are so many targets for me in the future – Object Detection,Pose estimation, Semantic segmentation,etc.. but also training my own model, tuning.

I hope all of you will like this tutorial. Thanks.

Footer_Basic

Please Support some devices for my blog

Amazon Gift List

Find ME

Twitter:@3threes2
Email:soup01threes*gmail.com (* to @)
YoutubeChannel:https://www.youtube.com/channel/UCQ3CHGAIXZAbeOC_9mjQiWQ

シェアする

  • このエントリーをはてなブックマークに追加

フォローする