From f059736e2d1f8422630b6860032e357a3447feed Mon Sep 17 00:00:00 2001 From: rubberhead Date: Wed, 13 Dec 2023 01:00:06 +0000 Subject: [PATCH] More work, ah well --- scratch/consistency-coherence.md | 48 +++++++++++++++++++++++++++++++ tex/misc/w12_slices.pdf | Bin 494222 -> 494226 bytes 2 files changed, 48 insertions(+) create mode 100644 scratch/consistency-coherence.md diff --git a/scratch/consistency-coherence.md b/scratch/consistency-coherence.md new file mode 100644 index 0000000..5c5ce8c --- /dev/null +++ b/scratch/consistency-coherence.md @@ -0,0 +1,48 @@ +# MRSW Coherence & Consistency Protocols +| | Sequential | TSO | PSO | Release | Acquire | Scope | +|------------------------|------------|-----|-----|---------|---------|-------| +| Home; Invalidate | [11,12,13] | | | [1,2] | [9] | [5] | +| Home; Update | | | | | | | +| Float Home; Invalidate | | | | [2] | | | +| Float Home; Update | | | | | | | +| Directory; Invalidate | [3] | | | | | | +| Directory; Update | | | | | | | +| Dist. Dir.; Invalidate | [6] | | [4] | [7] | [7,8] | | +| Dist. Dir.; Update | | | | [7] | | | + +Release consistency -- i.e., consistency between nodes are eagerly maintained (via invalidation or update). +Acquire consistency -- i.e., consistency between nodes are lazily maintained. + +In a home-based coherence protocol, the "home" node is guaranteed coherency with respect to the memory model i.e., the home node is always guaranteed to have the most recent rendition of data in its storage. Inversely, whoever have the most recent rendition of data in its storage can become the home node, which is the case for "floating home" protocols. + +In floating home protocols, some orchestration is needed for a stale node to be able to have its request message routed to the current home node. A simple implementation is to provide an orchestrator node which knows the order in which each node were "home" -- some garbage collection required. + +A scope-consistency protocol is one where each sharing is associated with a scope -- only loads and stores within the scope's participants is guaranteed to respect the memory model. + +Many newer systems don't even expose writable data movement (e.g., Grappa) -- to write, you have to move compute towards data. Reading is no-problem though. + +[7]'s case for acquire-invalidate holds only for MRSW (I think?). + +[10] writes an Itanium hypervisor to respect atomic and fence instructions. It's interesting, but because it emulates NUMA no coherence is applied between nodes -- there simply is no cache for remote nodes. + +[11] should be familiar. Home node also acts as the lock manager node (for simplicity). + +Home seems intuitive for implementing such a system, e.g., [12], which subconsciously used home-based DSM to implement their hypervisor mm. Their interests were in hypervisors, though. + +[14] is the Argo paper, which were compared with in [2]. Its coherence is entirely based on API function calls, which (ofc.) improves performance. Home-based. + +# References +[1] Shan, Tsai, and Zhang, "Distributed shared persistent memory", [10.1145/3127479.3128610](https://doi.org/10.1145/3127479.3128610) +[2] Endo, Sato, and Taura, "MENPS: A Decentralized Distributed Shared Memory Exploiting RDMA", [10.1109/IPDRM51949.2020.00006](https://doi.org/10.1109/IPDRM51949.2020.00006) +[3] Wang et al., "Concordia: Distributed shared memory with {In-Network} cache coherence", [fast'21](https://www.usenix.org/conference/fast21/presentation/wang) +[4] Cai et al., "Efficient distributed memory management with RDMA and caching", [10.14778/3236187.3236209](https://doi.org/10.14778/3236187.3236209) +[5] Hu et al., "JIAJIA: A Software DSM System Based on a New Cache Coherence Protocol", [10.1007/BFb0100607](https://doi.org/10.1007/BFb0100607) +[6] Chaiken et al., "LimitLESS Directories: A Scalable Cache Coherence Scheme", [10.1145/106975.106995](https://dl.acm.org/doi/pdf/10.1145/106975.106995) +[7] Carter, Bennett, and Zwaenepoel, "Implementation and Performance of Munin", [10.1145/121133.121159](https://doi.org/10.1145/121133.121159) +[8] Keleher et al., "Treadmarks: Distributed Shared Memory on Standard Workstations and Operating Systems", [EPFL](https://infoscience.epfl.ch/record/55805) +[9] Holsapple., "DSM64: A Distributed Shared Memory System in User-space", [CalPoly](https://doi.org/10.15368/theses.2012.40) +[10] Chapman and Heiser, "vNUMA: A Virtual Shared-Memory Multiprocessor", [usenix'09](https://www.usenix.org/legacy/event/usenix09/tech/full_papers/chapman/chapman.pdf) +[11] Kim et al., "DeX: Scaling Applications Beyond Machine Boundaries", [10.1109/ICDCS47774.2020.00021](https://doi.org/10.1109/ICDCS47774.2020.00021) +[12] Zhang et al., "GiantVM: a type-II hypervisor implementing many-to-one virtualization", [10.1145/3381052.3381324](https://doi.org/10.1145/3381052.3381324) +[13] Ding, "vDSM: Distributed Shared Memory in Virtualized Environments", [10.1109/ICSESS.2018.8663720](https://doi.org/10.1109/ICSESS.2018.8663720) +[14] Kaxiras, "Turning Centralized Coherence and Distributed Critical-Section Execution on their Head: A New Approach for Scalable Distributed Shared Memory", [10.1145/2749246.2749250](https://doi.org/10.1145/2749246.2749250) diff --git a/tex/misc/w12_slices.pdf b/tex/misc/w12_slices.pdf index ea552baa5fe180a5e0233ea775dc9c20b67a5d15..cfaa902ed4342371040c4c62a27bf405e4e41498 100644 GIT binary patch delta 2415 zcma*lc{tSj9tUuSv5P6oklkP?%rGOyax6m)nL)@>c3B#RnMq~)S+hlqZQ>LY_edmk z%2L+JUY6|35o3vvqjV$D)pO2$?sK1W@9jRX=lwjN=l%Kq@%`hIkt;*XmC00M*Ue{G zaH|5Nj#i{;gh2D#^S`gKr72t1%IH@`uz|U@vxaX-W5%udQ9T7B&@Y>(dl~D>ZDaRb z>%{zF(aNeHO5ixA04h>y^&unxD@o!R@@TRh8uN&=^*5+=sfS8$LiZordI}a6O^s9+ zwCPZ52YOFWe0?z?@tYZ*&V`q4bE+VkRs+2qY_B4z93|306ALuCLNCFeVyuJZ)E2Vw z+lHmZlZ^B>7sR^T_{()wDdq}Yy(In>fqXIYvv8{B9@TPK*6*F2C1iOq9dfR7?Vh)T zb&1j?3&ThEIX5+!)a07_vA zmh5mTn@xc*v>xy8EQT&zut@D)yYK)Xp%D{e`Q^6e6fr{1XDP$dxisl+`Iu(y>ydg@ zP6D|XshoP2q-HZAy7H~8mY@>+8uFGhD}mX(8U5`{KZ}}4Y00>z@R1=d79rHQ2Oq-# z22m~n`$cLSCG|VL`?A=HzG)aR;5PNUsns5}{XkiQT+%(%qLS=o)i`^3J2FLS-1u?Ml!;DYG(Vi3Y1(PM*nu zbE!c$S=zk1aBPx#rcqE5dNEG(JkG$&2Z@ zUUZaeY$J<2Cn7toB0LG}?-h8g=AwHR7Un~2acR7J1Um4!k1{Bq zs)D9=Po=^gHy%)umqiH|eEk7~?vLJGpNEGCaRZaEcnU`ng@gStpu>EhIS&Iq3|Ae6 zgF8Gz38{tDL8DPRE~Tvpw6qUbe=h5e0^HGHJCqg@g+QBX>zSZ* z%urZ83lu`nL{Hn)%)$&|Zi+-0$r=3c$wfZRv%bN8RDU_7_9=9}i48X#kRIzJh+Y+G z-O|gy9RcRV%JfN|_(LefLqQP+Pc}Hu`g-CYNU_EYP zISkNRu_TR^ls$!AoCbhnjv%G`aUxo9Gw_%Rdc;`0G6{1(iEDx_y&6(bEa6gYO)Hgu zN=&q_zRhgnsI$Taz@8ek_CXx?7#tV||5M{r+s2y2#Gpa${c7p3Ry;K3V)gOAx4KPm zb2Bc|9tg2^xb*S_61*GuKkv+eDtH~=&I=1LJ#|5+=b=(b#{gmAB@UYS{d*o#{jJWr z+%;XW3vu$ez{Z6^4Kn8mDHnl22ijQ1&YV_dt7tvnaERa~53%Jr%u6O{CdRy;x6Xq% zZ(e66_E~GnVDcqPS|?a%{N695=v0ToQYN#;tJ%`7H%kqt`q6=A_gBEbDpNbq5UqQl zsH=J7XNE(+u)M0TI(WTe{Xyxi96B0DY)*hRKR=8|*% zD->{G+-uuwdvugP1Ohh|O8a^cx^AU}=?Hjmrm-yd^UvjKP4NH8f7_Ja*eg?G>w{EZ zJs_}rQ1?c?`Hf!-AfF$_5@XB$LxORiX~-5is+FH57c9!=2X^(QgF zdF?XXKgniM57Mx&AEg0%+g(+0Zz|_@nLA=$pKNj8WNzh`!S>6PvMRkvChnBB3l3Yc zi%1(&@_+Ebp;@juagE{@r*=j@UdvuC+&PoD+Om@#xRSNL|7weDIAUUmYUT1zNFM|^ zUk!pDR`b?hUkYkPt8B*LSK`zM1zj1Awc~Ct@|s4PGe(iRd{Ki!)1KVx3G5T#XP!dq ziBZ+E_3q2@A;R@zOGzHkBTR@c`!uXCpM}U^qIG#oS{1SVidZAAg%Ga(c0GCwPFH9K zLJ79qRicBcLUzC3F@j%_*0c+l5sjGU)m-#Eqvl3t11-t+tnTbIhsfd45eJ$=DPYF-jGg))!A66wGl1OMQRC?k_ z&)rCo!r>dpVDA|*A>%E9_WJZt*O@_R-S@`gypH1JaMezpb5FFMc!i#-UsO$_#?rxg zFSO=T;tuIouxiU5-0@WD*U6+Bmz)n*fz221e#t2UWGh_la4OzUV7amHoz$&-S)mHK z{O+P>o8a_W2Sh}f$3Z~b6uE7>NbLES(zyedw&RLmq14Ogmgd)ZrR~iGv0`hCBkvgO zI>w5`1ZDplV&%_2`EEIE3Mzf$D4OqF;PVakp-{=bouu< M^C>Br6D;`t0qCYXv;Y7A delta 2372 zcma*d2{aV!9>;Op7{(ykw`>gtW1H+dGgC8U7?drHCA?f@iy>nfCQCvX;mwxxHr^~{ zP)KNO*(3X|c!?B3%IiJ%bkDi>y!Up`=X}q3&hP(Z<(x^+Ig|W;OBXJ32R=`>opTh#QNeW2}AjXXC#5pJr0GOAP0^4LP5694((MNX=)znR?Erf5>1 z484EtOmLgm{0>iAU5_ja)tLw$e0^R^b+RVOAe&`^&u^b`BbeD*xG1!AOA63O(q#2} zsoMOwv9=-zT;{Qt#-(|C`Pu%a6rbz6P|V0}co?*tEyUfUgD>)mf8`^zqp31ObB}A3 zQJV0}xH2J5aY)PD+rUhiIlr}o=;|&zU`GK^(7;4+-HuP;yLz@O6G7}jpQ65&Wkkdv zpKzDr9Y#0OMPUk6AynPsWiP&+#I*Ax>Kw1jCN3z2sYf+R*Z$Zg?#H z+D!Bg-73CcFEDl1%AhoyrhIc+pJWm$d>_g-d@;JeRHp`McbU}QIZKG=6=IB!9+h@Bt*n>P*nY^ zwjYRnf3SmX*(K}D30M#(DF%{QoldjT0fl$h)Rv=DhPf|ECRaq5x0=pYkmCLkb>Ck* z(IW~dUDV-l7k}v$h@9Nx&wp8XQ_&feDuchfqt5s$F1SM^KE zt;K7?HT@=JZGj$@{dC?+5kliKg%)CHuRL~neJXrtI;^$AOI9zis?)WCtoNIBMEYB< z-%-mSGiXn2<$B`AB^Nt;zKjfEtAZyO62h3tVa$vtF`e|+o8=_HNv!o@%s@gEL{(E= zRYL==;|K+*TnzL?;6uG6qLM&QGBd7kUxyCbz6sWH_F%C z>9E-9FgBNI<)UvS#Wa!+1i%#Sus=27Rg*2-&fEv-)T+`ESA=*4W$7IqMy-h)mzBXH zw~|G9gU=0-geVnqjxqa~F-SPOgvqZ2^r(b&%mntv#GsPQ>O+$CN#Yz-K;tBwXsxL6j)mFy z6&y|^4aWT`IAfI2Cn0<_)^eQ5sf6PhUUa85e4J6QL@@kiz&LXhUSfl6R_&0b!M(VD zsGSQYc{Flf-*?L%Wuuy_GLRS8#1cJ%#4KCUH;EDiUoj`#EW&avZ5(#~-) z=tp{sgUTT+ieir7U+l7S$M)wHDqA%gxR!#LW!+9ca1*)eCJ@jj_PU)(sqM5o9}{qf4(ZqcJ)-2-K?ugZn$ge~lL~l8&bjR;c6G;od$u zaKxKu?1z!Jd8s#yFJOMIhH$FObP5?mO(~P+hrpYAY6=Ap6qC90s&MK59i5waU#wgV zVlAH7Ys^DHEPugzxGD^(S=u0noBNvHTiC53|F<&1wl{WrH*ZO&UPE(73#gcjv2I0D zJr#>Oqjna5j)Y$GT*zMf?S6vFtK5P(Dpoq}-zeQHmT!}9TiiU)gZd`|ii85ABvC+`P3>*Nv_9Wp^