SuperPuperPreza

1.

Optimization of Computer
Vision Neural Network Layers
on RISC-V Architecture
Low-Level Vectorization and JIT Compilation of Math Kernels
in the OpenVINO Framework
Mentors:
Lecturer at the HPCSP Department of the ITMM Institute
Obolensky A.
Lecturer at the HPCSP Department of the ITMM Institute
Nesterov A.
Completed by:
ITMM Institute student gr. 3824B1PM2
Zhulin E.
ITMM Institute student gr. 3824B1FI1
Strelkov K.

2. Computer Vision

https://www.superannotate.com/blog/image-segmentation-for-machine-learning

3. Example: YADRO SmartFab

https://yadro.com/

4. YOLO: You Only Look Once

YOLOv8n-det
https://www.oejournal.org/
https://www.ultralytics.com/
YOLO: You Only Look Once

5. Datasets: COCO + ImageNet

https://cocodataset.org/
https://www.image-net.org/

6. OpenVINO Open Visual Inference and Neural Network Optimization

https://docs.openvino.ai/

7. Hardware: RISC-V

https://roalogic.github.io/RV12
https://riscv.org/

8. YOLO + OpenVINO + RISC-V = Takes too long!

Latency (sec)
800
700
600
500
400
300
200
100
0
YOLOv5n-det YOLOv8n-det YOLO11n-det YOLO26n-det
*benchmark_app – Benchmark Tool to estimate deep learning inference performance in OpenVINO framework

9. Why is it take too long?

YOLOv8n-det
Transpose
0%
Latency (ms)
Concat Quantity
7%
Concat
4,692
Convolution Transpose
27%
Subgraph
Subgraph
30%
28,704
61,384
Softmax
394
Reorder
9,785
MaxPool
15,855
Interpolate
3,42
SiLU
Softmax
0%
Reorder
10%
MaxPool
1%
Interpolate
1%
SiLU
24%
3226,914
750216,498
Convolution
0
200000
400000
600000
800000

10. Convolution: naive

Pros:
+ simple to implement
+ versatility for any platform
Cons:
– irregular memory access
– cache eviction
– cache misses
https://hannibunny.github.io/mlbook/neuralnetworks/
https://toast-lab.sist.shanghaitech.edu.cn/courses/CS110@ShanghaiTech/Spring-2024/project/p1.2-web/

11. Convolution: im2col

Pros:
+ cache locality
+ GEMM
+ reduced loop overhead
+ high hardware utilization
Cons:
– higher memory overhead
– implementation complexity
https://www.researchgate.net/publication/332186100_DeLTA_GPU_Performance_Model_for_Deep_Learning_Applications_with_In-depth_Memory_System_Traffic_Analysis

12. Convolution: naive vs im2col

13. RVV (RISC-V Vector Instruction)

14. SiLU

https://statisticaloddsandends.wordpress.com/2023/10/25/what-is-the-swish-activation-function/
https://www.researchgate.net/figure/aReLU-and-Swish-Functions-bDerivative-of-ReLU-and-Swish_fig1_325332854

15. Just-in-time

16. YOLO + OpenVINO + RISC-V + We = High Performance!

Latency (ms)
35000
30000
25000
20000
15000
10000
5000
0
YOLOv5
YOLOv8
n-det
s-det
YOLO11
m-det
n-cls
s-cls
m-cls
YOLO26

17. YOLO + OpenVINO + RISC-V + We = High Performance!

FPS
2,5
2
1,5
1
0,5
0
YOLOv5
YOLOv8
n-det
YOLO11
s-det
m-det
YOLO26

18. Accuracy Validation

mAPval (50-95)
45
Top1 & Top5
100
90
80
70
60
50
40
30
20
10
0
40
35
30
25
20
15
10
5
0
YOLOv5n-det YOLOv8n-det YOLO11n-det YOLO26n-det
Standard
Current
YOLOv5n-cls
YOLOv8n-cls
YOLO11n-cls
YOLO26n-cls
Standard(Top1)
Current(Top1)
Standard(Top5)
Current(Top5)

19. Real Deploy Result

20. Integration

https://github.com/openvinotoolkit/openvino/pull/35852
https://github.com/openvinotoolkit/openvino/pull/35949

21. Project results & Development Roadmap

Project results & Development Roadmap
Latency (sec)
800
753,962028
700
600
518,252327
500
434,012083
400
Winograd convolution
300
Depth wise convolution
SoftMax function
And other latest layers from VLM…
200
100
96,21531
0,51175
4,47696
3,91054
3,2109
0
YOLOv5n-det YOLOv8n-det YOLO11n-det YOLO26n-det
before
after

22. Thx you for your attention