Use Intel VNNI for int dot product

[amitdo]

There are two variants:

• AVX512_VNNI (Tiger Lake, Rocket Lake) - 512bit/256bit/128bit
• AVX_VNNI - (upcoming Alder Lake) - 256bit/128bit

VNNI replaces 3 simd instructions with one instruction.

It seems that we can use it inside MultiplyGroup().

https://software.intel.com/content/www/us/en/develop/articles/intel-advanced-vector-extensions-512-intel-avx-512-new-vector-neural-network-instruction.html

[stweil]

That requires access to test machines which support those instructions. Up to now I don't have any server with AVX512.

[amitdo]

AVX_VNNI - (upcoming Alder Lake) - 256bit/128bit

This will be available soon. Maybe we could ask users to help testing this feature a few months from the launch.

[amitdo]

@stweil, can I add AVX_VNNI (256 bit) detection as a first step?

[stweil]

Sure, but who has such hardware to test it?

[amitdo]

There are tens of millions of people with Intel's Alder Lake, some of them are Tesseract users. We can ask in the forum to test the detection (and later the intdotproductvnni). Hopefully, we will find at least one person that has this CPU and is willing to help.

[stweil]

@amitdo, I just noticed that the notebook which I used for AVX512F also has AVX512VNNI. :-)
Do you already have code for the detection? If not, I can add it (just have to find the right documentation which bit in cpuid is used). Wikipedia has the required documentation.

[amitdo]

Go ahead!

Could you do the AVX_VNNI detection too? You are more familiar with the detection code than me.

Since most of our files already have:

(C) Copyright <year>, Google Inc.

I think we can look at other Google projects with the same license as ours, and use parts of the code if we need it.

google/cpu_features#263

https://github.com/tensorflow/tensorflow/blob/18d203b1ef84b1e2d4de9eb249194ab1386bdd7b/tensorflow/core/platform/cpu_info.cc

[stweil]

Detection is now implemented by commit 0daf18c.

[amitdo]

I see that you check that avx/avx2 is supported by the OS. Do you also check somewhere that avx512 is supported by the OS?

[stweil]

No, currently only the hardware capabilities are checked for avx512. Up to now nobody complained, so maybe AVX512F was only used on operating systems which support it. I'll add a check for OS support. Thank you for the hint!

[amitdo]

I will try to implement intsimdmatrixavx512vnni.cpp.