[libc++] Make std::align an inline function

[poor-circle]

std::align is heavily used in memory allocators. When we attempted to switch from libstdc++ to libc++, we observed a 50% performance regression in a database query bench: the issue is that std::align in libc++ is not an inline function, which prevents the compiler from performing inlining optimizations.

make std::align an inline function will run about 2x faster. See benchmark result.

[poor-circle]

@ChuanqiXu9

[llvmbot]

@llvm/pr-subscribers-libcxx

Author: saipubw (poor-circle)

Changes

std::align is heavily used in memory allocators. When we attempted to switch from libstdc++ to libc++, we observed a performance regression: the issue is that std::align in libc++ is not an inline function, which prevents the compiler from performing inlining optimizations. make std::align an inline function will run about 3x faster.

benchmark result:

---

Full diff: https://github.com/llvm/llvm-project/pull/167472.diff

2 Files Affected:

• (modified) libcxx/include/__memory/align.h (+19)
• (modified) libcxx/src/memory.cpp (+3)

diff --git a/libcxx/include/__memory/align.h b/libcxx/include/__memory/align.h
index 402eac3380925..80e504fc1dfb7 100644
--- a/libcxx/include/__memory/align.h
+++ b/libcxx/include/__memory/align.h
@@ -11,6 +11,7 @@
 
 #include <__config>
 #include <__cstddef/size_t.h>
+#include <cstdint>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
@@ -18,7 +19,25 @@
 
 _LIBCPP_BEGIN_NAMESPACE_STD
 
+// From >=v2 ABI, std::align is an inline function.
+#if _LIBCPP_ABI_VERSION >= 2
+inline _LIBCPP_HIDE_FROM_ABI void* align(size_t __align, size_t __sz, void*& __ptr, size_t& __space) {
+  void* r = nullptr;
+  if (__sz <= __space) {
+    char* p1 = static_cast<char*>(__ptr);
+    char* p2 = reinterpret_cast<char*>(reinterpret_cast<uintptr_t>(p1 + (__align - 1)) & -__align);
+    size_t d = static_cast<size_t>(p2 - p1);
+    if (d <= __space - __sz) {
+      r     = p2;
+      __ptr = r;
+      __space -= d;
+    }
+  }
+  return r;
+}
+#else
 _LIBCPP_EXPORTED_FROM_ABI void* align(size_t __align, size_t __sz, void*& __ptr, size_t& __space);
+#endif
 
 _LIBCPP_END_NAMESPACE_STD
 
diff --git a/libcxx/src/memory.cpp b/libcxx/src/memory.cpp
index 9be40cb9c1285..9efbb6eb8eca3 100644
--- a/libcxx/src/memory.cpp
+++ b/libcxx/src/memory.cpp
@@ -132,6 +132,8 @@ __sp_mut& __get_sp_mut(const void* p) {
 
 #endif // _LIBCPP_HAS_THREADS
 
+// Remove std::align from >=v2 dylib ABI, make it an inline function.
+#if _LIBCPP_ABI_VERSION == 1
 void* align(size_t alignment, size_t size, void*& ptr, size_t& space) {
   void* r = nullptr;
   if (size <= space) {
@@ -146,5 +148,6 @@ void* align(size_t alignment, size_t size, void*& ptr, size_t& space) {
   }
   return r;
 }
+#endif
 
 _LIBCPP_END_NAMESPACE_STD

[yuxuanchen1997]

This also needs release notes and changes to libcxx/docs/ABIGuarantees.rst. Example: e8fa13c

[ldionne]

This is a simple function, I do wonder why it was written inside the dylib in the first place. I think it would be reasonable to make this inlineable.