A Solution Type Indexed Serialization Combinators in Standard ML

Advanced Topics in Language-based Security,Spring2005Lecture9,Thursday May26,2005Type-Specialized Serialization with Sharing•Motivation•T ype indexed serialization combinators in Standard ML•Using the combinators:basic combinators,enumeration datatypes,single recursive datatypes,references•On programmer control and type safety•Implementation issues:type dynamic,a stream module,basic combinators,references, sharing,datatypes•Related work•Possible future workAdvanced T opics in Language-based Security,Spring2005IT University of Copenhagen Page9-1MotivationReasons for programming language serialization support:•Interchange of values for distributed computing•Saving program state to disk•Probably other stuff...A Solution:Type Indexed Serialization Combinators in Standard MLsignature PICKLE=sigtype’a pu(*abstract pickle/unpickle type*)val pickle:’a pu->’a->stringval unpickle:’a pu->string->’aval word:word puval string:string puval real:real puval pair:’a pu*’b pu->(’a*’b)puval triple:’a pu*’b pu*’c pu->(’a*’b*’c)puval list:’a pu->’a list puval option:’a pu->’a option puendstructure Pickle:>PICKLE=...Advanced T opics in Language-based Security,Spring2005IT University of Copenhagen Page9-3Combinator Solution Features•Compactness due to specialization.No type information(tagging)is exported by the serializer,which leads to compact serialized data.All necessary type information for deserializing the serialized valueis present in the type specialized deserializer.•Compactness due to sharing.Serialization of two equivalent values leads to sharing in both serialized and deserialized data.•Type safety.A type specialized serializer may be applied only to values of the specialized type.•Programmer control.The programmer may exploit knowledge about data invariants to obtain efficient serializers in cases where generic serialization does not perform well.•No need for runtime system support.The combinator library imposes no restrictions on the representation of values.In particular,the technique supports a tag-free representation of values,asthe library is written entirely in the language itself.Example:A serializer for lists of pairs of integers•By combining the int,pair,and list combinators,it is possible to construct a serializer for integer-pair lists:val pu_pairs:(int*int)list pu=let open Pickle in list(pair(int,int))endval s:string=Pickle.pickle pu_pairs[(2,3),(1,2),(2,3)]•Sharing is introduced by the serializer,which means that when the serialized string isdeserialized into a list,the pairs(2,3)in the deserialized list share the samerepresentation:val l:(int*int)list=Pickle.unpickle pu_pairs sAdvanced T opics in Language-based Security,Spring2005IT University of Copenhagen Page9-5Serializing Enumeration Datatypes•The combinator library contains the following combinator for building serializers and deserializers for“enumeration”datatypes:val enum:(’a->int)*’a list->’a pu•Given a datatype with n nullary value constructors C0···C n−1,a serializer-deserializerpair may be constructed by passing to the enum combinator(1)a function mapping eachconstructor C i to the integer i,where0≤i<n,and(2)the list[C0,···,C n−1].Example:•For constructing a serializer for the datatypedatatype color=R|G|Bcode is written as follows:val pu_color:color Pickle.pu=Pickle.enum(fn R=>0|G=>1|B=>2,[R,G,B])Serializing Single Recursive Datatypes•The following combinators allow for building serializers and deserializers for“single recursive”datatypes:val data:(’a->int)*(’a pu->’a pu)list->’a puval con0:’a->’b->’a puval con1:(’a->’b)->(’b->’a)->’a pu->’b pu•Given a datatype t with n value constructors C0···C n−1,a serializer-deserializer pairfor t may be constructed by passing to the data combinator,(1)a function mapping avalue constructed using C i to the integer i,where0≤i<n,and(2)a list of functions[f0,···,f n−1],where each function f i,0≤i<n,is a serializer for the datatype forthe constructor C i,parameterized over a serializer to use for recursive instances of t. Advanced T opics in Language-based Security,Spring2005IT University of Copenhagen Page9-7Example:Building a serializer-deserializer pair for a tree datatype•Consider the datatype:datatype tree=L|N of tree*int*tree•T o construct a serializer for the datatype tree,the data combinator can be applied,together with the utility functions con0and con1:val pu_tree:tree Pickle.pu=let open Picklefun pu_L pu=con0L pufun pu_N pu=con1N(fn N a=>a)(triple(pu,int,pu))in data(fn L=>0|N_=>1,[pu_L,pu_N])endAdvanced T opics in Language-based Security,Spring2005IT University of Copenhagen Page9-9Serializers for References•The following combinators are supported for references:val refCyc:’a->’a pu->’a ref puval ref0:’a pu->’a ref puval refLin:’a pu->’a ref pu•The combinator ref0assumes that the reference—when serialized—does not contribute to a cycle in the value.•The combinator RefCyc takes as itsfirst argument a dummy value for the type of the reference content,which allows the deserializer to reintroduce cycles appearing in theoriginal value.•The refLin combinator assumes that,for each of the reference values,there is only ever one pointer to the reference.On Programmer Control•The programmer may exploit knowledge about data invariants to obtain efficient serializers in cases where generic serialization does not perform well.•In particular,the implementation based on hash-consing does not always perform well:–e.g.,for serializing many values of type bool ref in cases where each value is usedlinearly;that is,with only one pointer to it.Advanced T opics in Language-based Security,Spring2005IT University of Copenhagen Page9-11On Type Safety(for a combinator subset)•A type specialized serializer may be applied only to values of the specialized type.•When a value is deserialized,a type checksum in the serialized data can be checked against the type checksum of the specialized deserializer.•A subset of the library is truly type safe in the sense that with this subset,it is not possible to construct serializers that do not behave as expected.Type Safety PropertiesLemma(Type preservation)Assume T is a type constructed using the Pickle combinators.If v:|T|and s=pickle T v then there exists v such that v =unpickle T sand v :|T|.Lemma(Semantic Soundness)Assume T is a type constructed using the Pickle combinators.If v:|T|and s=pickle T v then there exists v such thatv =unpickle T s and v =v.Implementation overview•The implementation of the combinator library makes use of1.An implementation of type dynamic in Standard ML2.A stream implementation.3.A hash table implementation.Advanced T opics in Language-based Security,Spring2005IT University of Copenhagen Page9-13An implementation of type dynamic•Here is a signature for a type dynamic with hash function support end equality support.signature DYN=sigtype dynval new:(’a*’a->bool)->(’a->word)->(’a->dyn)*(dyn->’a)val eq:dyn*dyn->boolval hash:dyn->wordend•Efficient hash function support and efficient equality function support is not possible by extending the folklore implementation of type dynamic in Standard ML using exceptions.•Efficient hash-function support and efficient equality support is possible with an implementation based on references and higher-order functions.An implementation of type dynamic using references and higher-order functionsstructure Dyn:>DYN=structdatatype method=RESET|EQ|SET|HASHtype dyn=method->wordfun new eq h=let val r=ref NONEin(fn x=>fn HASH=>h x|RESET=>(r:=NONE;0w0)|SET=>(r:=SOME x;0w0)|EQ=>case!r of NONE=>0w0|SOME y=>if eq(x,y)then0w1else0w0,fn f=>(r:=NONE;f SET;valOf(!r)))endfun eq(f1,f2)=(f2RESET;f1SET;f2EQ=0w1)fun hash f=f HASHendAdvanced T opics in Language-based Security,Spring2005IT University of Copenhagen Page9-15Input and output streamssignature STREAM=sigtype IN and OUT(*kinds*)type’k streamval getLoc:’k stream->wordval out:char*OUT stream->OUT streamval get:IN stream->char*IN streamval outw:word*OUT stream->OUT streamval getw:IN stream->word*IN streamval outcw:word*OUT stream->OUT streamval getcw:IN stream->word*IN streamval toString:OUT stream->stringval openOut:unit->OUT streamval openIn:string->IN streamendHash tablessignature POLYHASH=sigtype(’key,’data)hash_tableval mkTable:(’key->int)*(’key*’key->bool)->int*exn->(’key,’data)hash_table val insert:(’key,’data)hash_table->’key*’data->unitval peek:(’key,’data)hash_table->’key->’data optionendAdvanced T opics in Language-based Security,Spring2005IT University of Copenhagen Page9-17Representing Serializers•The abstract type’a pu is defined by the following type declarations:type pe=(Dyn.dyn,S.loc)H.hash_tabletype upe=(S.loc,Dyn.dyn)H.hash_tabletype instream=S.IN S.stream*upetype outstream=S.OUT S.stream*petype’a pu={pickler:’a->outstream->outstream,unpickler:instream->’a*instream,hasher:’a->int*word->int*word,eq:’a*’a->bool}•The hasher takes a“current depth”and an accumulator as argument and returns a new current depth and an updated accumulator.Hash Combinators•Here are some hash combinators that are used for implementing the serialization combinators:local val Alpha=0w65599val Beta=0w19in val maxDepth=50fun hashAdd w(a,d)=(w+a*Alpha,d-1)fun hashAddSmall w(a,d)=(w+a*Beta,d-1)fun maybestop f((a,d)as s)=if d<=0then s else f send•T o avoid that values are traversed fully by the constructed hash functions and to ensure termination of hash functions in case of cycles,the combinators count the number of hashoperations performed by the hash function.Advanced T opics in Language-based Security,Spring2005IT University of Copenhagen Page9-19A Serialization Combinator for Word Valuesval word:word pu={pickler=fn w=>fn(s,pe)=>(S.outw(w,s),pe),unpickler=fn(s,upe)=>let val(w,s)=S.getw sin(w,(s,upe))end,hasher=hashAdd,eq=op=}A Serialization Combinator for Pairsfun pair(pu1:’a pu,pu2:’b pu):(’a*’b)pu={pickler=fn(v1,v2)=>fn s=>let val s=#pickler pu1v1sin#pickler pu2v2send,unpickler=fn s=>let val(v1,s)=#unpickler pu1sval(v2,s)=#unpickler pu2sin((v1,v2),s)end,hasher=fn(v1,v2)=>maybestop(fn s=>#hasher pu2v2(#hasher pu1v1s)),eq=fn((a1,a2),(b1,b2))=>#eq pu1(a1,b1)andalso#eq pu2(a2,b2)}Advanced T opics in Language-based Security,Spring2005IT University of Copenhagen Page9-21A Sharing Combinator•We shall now see how it is possible to construct a combinator share that leads to sharing of serialized and deserialized data.Serialization•The share serializerfirst checks if some identical value is associated with a location l in the pickle environment.In this case,a REF-tag is written to the outstream together with areference to the location l.•If there is no value in the pickle environment identical to the value to be serialized,aDEF-tag is written to the output stream,the current location l of the output stream isrecorded,the value is serialized,and an entry is added to the pickle environment mappingthe value into the location l.A Sharing Combinator(cont.)Deserialization•When a value is deserialized by the share combinator,first the tag(i.e.,REF or DEF)is read from the input stream.•If the tag is a REF-tag,a location l is read from the input stream,and the location l is used for looking up a resulting value for the deserializer in the unpickler environment.•If,on the other hand,the tag is a DEF-tag,the location l of the input stream is recorded,avalue v is deserialized with the argument deserializer,andfinally,an entry is added to theunpickler environment mapping the location l into the value v,which is also the result ofthe deserialization.Notice•Efficient inhomogeneous environments,mapping values of different types into locations,are possible only through the use of the particular Dyn library,which supports a hashfunction on values of type dyn and an equality function on values of type dyn.Advanced T opics in Language-based Security,Spring2005IT University of Copenhagen Page9-23Implementation of the share Combinator(part1)fun share(pu:’a pu):’a pu=let val REF=0w0and DEF=0w1val(toDyn,fromDyn)=Dyn.new(#eq pu)(fn v=>#1(#hasher pu v(0w0,maxDepth)))in{pickler=fn v=>fn(s,pe)=>let val d=toDyn vin case H.peek pe d ofSOME loc=>let val s=S.outcw(REF,s)val s=S.outw(loc,s)in(s,pe)end|NONE=>let val s=S.outcw(DEF,s)val loc=S.getLoc sval res=#pickler pu v(s,pe)in case H.peek pe d of SOME_=>res|NONE=>(H.insert pe(d,loc);res)endend,Implementation of the share Combinator(part2) unpickler=fn(s,upe)=>let val(tag,s)=S.getcw sin if tag=REF thenlet val(loc,s)=S.getw sin case H.peek upe loc ofSOME d=>(fromDyn d,(s,upe))|NONE=>fail"share.error"endelse(*tag=DEF*)let val loc=S.getLoc sval(v,(s,upe))=#unpickler pu(s,upe)in H.insert upe(loc,toDyn v);(v,(s,upe))endend,hasher=fn v=>maybestop(#hasher pu v),eq=#eq pu}endAdvanced T opics in Language-based Security,Spring2005IT University of Copenhagen Page9-25A Combinator for Serializing Cyclic Data•In Standard ML,all cycles goes through a ref cell.Serialization•Thefirst time a reference value is serialized,a DEF-tag is written to the current location l of the outstream.•Thereafter,the pickle environment is extended to associate the reference value with thelocation l.•Then the argument to the reference constructor is serialized.•If,on the other hand,it is recognized that the reference value has been serialized earlier(i.e.,byfinding an entry in the pickle environment mapping the reference value to a streamlocation l),a REF-tag is written to the outstream,followed by the location l.A Combinator for Serializing Cyclic Data(cont.)Deserialization•First the location l of the input stream is obtained.•Second,a reference value r is created with the argument being the dummy given asargument to the refCyc combinator.•Then the unpickle environment is extended to map the location l to the reference value r.•Thereafter,a value is deserialized,which is then assigned to the reference value r.This assignment establishes the cycle and the dummy value no longer appears in thedeserialized value.Advanced T opics in Language-based Security,Spring2005IT University of Copenhagen Page9-27Implementation of the refCyc Combinator(part1)fun refCyc(dummy:’a)(pu:’a pu):’a ref pu=let val REF=0w0and DEF=0w1fun hasher(ref v)=maybestop(fn p=>#hasher pu v p)val(toDyn,fromDyn)=Dyn.new(op=)(fn v=>#1(hasher v(0w0,maxDepth)))in{pickler=fn r as ref v=>fn(s,pe)=>let val d=toDyn rin case H.peek pe d ofSOME loc=>let val s=S.outcw(REF,s)val s=S.outw(loc,s)in(s,pe)end|NONE=>let val s=S.outcw(DEF,s)val loc=S.getLoc sin H.insert pe(d,loc);#pickler pu v(s,pe)endend,Implementation of the refCyc Combinator(part2)unpickler=fn(s,upe)=>let val(tag,s)=S.getcw sin if tag=REF thenlet val(loc,s)=S.getw sin case H.peek upe loc ofSOME d=>(fromDyn d,(s,upe))|NONE=>fail"ref.error"endelse(*tag=DEF*)let val r=ref dummyval_=H.insert upe(S.getLoc s,toDyn r)val(v,(s,upe))=#unpickler pu(s,upe)in r:=v;(r,(s,upe))endend,hasher=hasher,eq=op=}endAdvanced T opics in Language-based Security,Spring2005IT University of Copenhagen Page9-29The data Combinatorfun data(toInt:’a->int,fs:(’a pu->’a pu)list):’a pu=let val res:’a pu option ref=ref NONEval ps:’a pu vector option ref=ref NONEfun p v(s,pe)=let val i=toInt vval s=S.outcw(Word.fromInt i,s)in#pickler(getPUPI i)v(s,pe)endand up(s,upe)=let val(w,s)=S.getcw sin#unpickler(getPUPI(Word.toInt w))(s,upe)endand eq(a1:’a,a2:’a):bool=toInt a1=toInt a2andalso#eq(getPUPI(toInt a1))(a1,a2) and getPUP()=case!res ofNONE=>let val pup=share{pickler=p,hasher=h,unpickler=up,eq=eq}in res:=SOME pup;pup end|SOME pup=>pupThe data Combinator(cont.)and getPUPI(i:int)=case!ps ofNONE=>let val ps0=map(fn f=>f(getPUP()))fsval psv=Vector.fromList ps0in ps:=SOME psv;Vector.sub(psv,i)end|SOME psv=>Vector.sub(psv,i)and h v=maybestop(fn p=>let val i=toInt vin hashAddSmall(Word.fromInt i)(#hasher(getPUPI i)v p)end)in getPUP()endAdvanced T opics in Language-based Security,Spring2005IT University of Copenhagen Page9-31Notes on the data Combinator•T o allow for arbitrary sharing between parts of a data structure(of some datatype)and perhaps parts of another data structure(of the same datatype),the combinator makes useof the share combinator.•It is essential that the share combinator is not only applied to the resulting serialization combinator for the datatype,but that the sharing version of the combinator is used forrecursive occurrences.Otherwise,it would not be possible to obtain sharing between thetail of a list and some other list being serialized.•Thus,the four functions(the pickler,unpickler,equality function,and hash function)that make up the serializer are mutually recursive and a caching mechanism(the functiongetPUP)makes sure that the share combinator is applied only once.Related Work•Many languages provide support for serializing values,including Modula-3,Java,and C#.For other languages,such as Standard ML,programmers have relied on implementationsupport for serialization(SML/NJ0.93has built-in support for serialization,for instance.)•Filinski recognizes a direct implementation of type dynamic in ML[Filinski’96,page106], which does not make use of the folklore implementation technique that uses exceptions toimplement an extensible datatype.•Andrew Kennedy’s similar combinator library[JFP’05].His work does not support serialization of cyclic data;nor does it support implicit sharing.•The Zephyr Abstract Syntax Description Language(ASDL)[DSL’97].•Packed Encoding Rules(PER)of ASN.1[X.691].Advanced T opics in Language-based Security,Spring2005IT University of Copenhagen Page9-33Future Work•Fixing performance issues using a variation of multi set discrimination.•Support for serialization of program code.–what kind of support is needed from implementersOpen Problem:An MSD Approach to Serializing References•Can a solution based on Multi Set Discrimination(MSD)solve the general performance problem related to hash consing of references.•Multi set discrimination requires runtime support.It requires a discriminator msdref,oftype’a ref list->’a ref list,that,given a list r=[r1,···,r n]ofreferences,returns(in O(n)time)the list of distinct references d in r with the propertythat removing all but thefirst occurrences of elements from r yields d.•Such a discriminator can be seen as a generalization of equality on references–given two references r and r ,the expression msdref[r,r ]returns a singleton listif the references are identical(equal in Standard ML)or a list containing two elements ifthe references are not identical.•But,perhaps another kind of discriminator is needed:val msdref’:’a ref list->(’a ref*int)listAdvanced T opics in Language-based Security,Spring2005IT University of Copenhagen Page9-35。