08 LinkPrediction

Transcript

1 Link Prediction Davide Mottin, Konstantina Lazaridou Hasso Plattner Institute Graph Mining course Winter Semester 2016

2 Acknowledgements § : Most of this lecture is taken from www.cs.uoi.gr l14/ tsap /teaching/cs - http:// /~ § Other adapted content is from: • Lu, L. and Zhou, T., 2011. Link prediction in complex networks: A survey. Physica A: Statistical Mechanics and its Applications , 390 (6), pp.1150 - 1170 . • Chouldechova : The link prediction problem for social networks, Alexandra http:// statweb.stanford.edu /~ owen /courses/319/ achouldechova.pdf GRAPH MINING WS 2016 2

3 Link Prediction Will nodes 33 and 28 become friends in the future? network Does the structure contain enough information to predict which new links What about formed in the will be nodes 27 and 4? future? GRAPH MINING WS 2016 3

4 Why link prediction? § new friends in online social networks . Recommending Recommending § pages to subscribe Predicting the participation of actors in events § Suggesting interactions between the members of a § company/organization that are external to the hierarchical structure of the organization itself . § Predicting connections between members of terrorist organizations who have not been directly observed to work together . § Suggesting collaborations between researchers based on co - authorship. § Overcoming the data - sparsity problem in recommender systems using collaborative filtering GRAPH MINING WS 2016 4

5 Who to follow GRAPH MINING WS 2016 5

6 Understanding the network Understanding how social networks evolve § § The link prediction problem • Given a snapshot of a social network at time 푡 , we seek to accurately predict the edges that will be added to the network during the interval ( 푡 , 푡’ ) ? ... time t t ’ GRAPH MINING WS 2016 6

7 Lecture road Unsupervised methods Classification approaches Who to follow GRAPH MINING WS 2016 7

8 Link prediction problem 퐺 let , 푡 < 푡 For [ 푡 , 푡 ] denote the subgraph of 퐺 consisting of edges all ( ( - between 푡 and 푡 . given For 푡 that 푡 took < 푡 < 푡 place ′ , < . . ( ( ( ′ not edges of list a output to wish 퐺 , ] ′ 푡 , 푡 [ 퐺 [ 푡 we , are that ] 푡 in ( ( ( ( appear , 퐺 [ to 푡 ] predicted ′ 푡 in . . ] training interval ü [ t , tʹ 0 0 , t [ ü test interval ] tʹ 1 1 the topology of the network on proximity) (the more § solely Based (social also considers attributes of general nodes and links ) the problem (hidden) from problem of inferring missing the links Different (there is § a temporal aspect) GRAPH MINING WS 2016 8

9 Link Prediction concepts - - 푡 < 푡 푡 푡 < < . ( . ( 1 1 3 2 2 3 4 4 - - 퐺 푡 푡 , 푡 퐺 , 푡 ( ( . . } 3 2 { = 퐸 , , 1 , 3 , 2 = , 퐸 4 1 } { , 2 ;<= 123 Definition [Core] 푉 퐶표푟푒 ⊂ edges in is the set of all nodes that are incident to at least 휅 EFGH;H;I - - 퐺 [ 푡 푡 , , 푡 ] ] and at least 휅 [ edges in 퐺 푡 E

10 An example for link prediction § - authorship network (G) from “author list” of the physics e - Co Print ( www.arxiv.org ) arXiv § Took 5 such networks from 5 sections of the print B is in the core D B B A C A C Test interval [1997,1999] Training interval [1994,1996] 풕풆풔풕 휿 = 휿 풕풓풂풊풏풊풏품 = ퟐ ퟐ papers during both training and test 2 : set of authors who have at least Core 퐺 [ 〉 퐸 푛푒푤 = new collaborations (edges) 1994 , 1996 ] = 퐺 푐표푙푙푎푏 = 〈 퐴 , 퐸 123 GRAPH MINING WS 2016 10

11 Data GRAPH MINING WS 2016 11

12 Example Dataset: co - authorship interval = , t ʹ t = 1996 : training 1994 - > [ 1994 , 1996 ] 0 0 ] = 1997 , t ʹ 1999 = 1999 : t interval - > [ 1997 , test 1 1 〉 퐺 = 〈 푉 , 퐸 • ] 1996 = 퐺 [ 1994 , 푐표푙푙푎푏 푑 표푙 • 퐸 the : authors in V that co - author a paper during the test interval but not during 푛푒푤 interval training • κ papers = 3 , κ 3 = 3 : Core consists of all authors who have written at least test training during the training period and at least 3 papers during the test period Predict 퐸 푛푒푤 GRAPH MINING WS 2016 12

13 Methods for link prediction § a connection weight score(x, y) to each pair of nodes Assign , based (x on the input graph y) Produce § ranked list of decreasing order of score a § We can consider all links incident to a specific node x , and recommend x the top ones to § If we focus to a specific x, the score can be seen as a centrality measure x for How to assign the score(x, y) between two nodes x and y? ü node proximity Some form of similarity or GRAPH MINING WS 2016 13

14 Lecture road Unsupervised methods Classification approaches Who to follow GRAPH MINING WS 2016 14

15 Summary of unsupervised methods Neighborhood based approaches § • Common neighbors, Adamic , Jaccard , ... § Path based approaches • Shortest path, Katz § Low - rank approximation § Clustering and mixed approaches GRAPH MINING WS 2016 15

16 LP Methods: Neighborhood - based Intuition The larger the overlap of the neighbors of two nodes, the more likely the nodes to be linked in the future i.e , Γ 푥 = 푦 푥 denote the set of nodes adjacent to 푦 ∈ 퐸 } Let 푥 Γ 푥 , , Common neighbors: § how many neighbors are in common between x and y = 푥 ∩ Γ 푦 푠푐표푟푒 Γ 푥 , 푦 how likely a neighbor of x is also a neighbor of y coefficient: § Jaccard 푦 Γ 푥 ∩ Γ = 푥 , 푦 푠푐표푟푒 Γ 푥 ∪ 푦 Γ /Adar: § large weight to common neighbors with low degree (the lower Adamic the degree the higher the relevance) 1 = 푠푐표푟푒 푥 , 푦 h | | 푧 Γ log ∩ p r o ∈ p q Adamic 1.4 Neighbors who are linked with 2 nodes are assigned weight = 1/log(2) = § § Neighbors who are linked with 5 nodes are assigned weight = 1/log(5) = 0.62 GRAPH MINING WS 2016 16

17 LP Methods: Preferential attachment Intuition The more popular a node is the more probable it will form a link with popular nodes Γ 푥 , i.e , Γ 푥 Let 푥 = denote the set of nodes adjacent to § ∈ 퐸 } 푦 , 푥 푦 푦 = Γ 푥 | | Γ 푦 푠푐표푟푒 푥 , free network formation § Inspired to scale - § evidence to suggest that co - Researchers found empirical product of the neighborhood authorship is correlated with the sizes This depends on the degrees of the nodes not on their neighbors per se GRAPH MINING WS 2016 17

18 Other neighborhood based methods p q ∩ p r § 푥 , 푦 = Salton index: 푠푐표푟푒 | r | q p p p s q ∩ p r index: 푠푐표푟푒 푥 Sørensen 푦 = § , q t | p r | p q p ∩ p r Hub Promoted Index: 푥 , 푦 = 푠푐표푟푒 § { p q , p r } uvw p q ∩ p r 푥 , 푦 = § Hub Depressed Index : 푠푐표푟푒 uxy p q , p r } { p q ∩ p r 푦 푥 , - = Leicht Holme § Newman Index: 푠푐표푟푒 - p q p r . ∑ Resource allocation: 푦 = § 푥 푠푐표푟푒 , o ∈ p q ∩ p r | p o | GRAPH MINING WS 2016 18

19 Methods for Link Prediction: Path based Intuition Use the (shortest) distance between two nodes as a link prediction measure − 푥 ∈ 푉 × 푉 푦 퐸 , For § 123 푠푐표푟푒 푥 , 푦 = (negated) length of shortest path between x and y 3 - s core( x,y ) = y Very basic approach, it does not consider connections among ( x,y ) but only the distance x GRAPH MINING WS 2016 19

20 LP Methods: Path based ) in the x,y Element ( Katz index § matrix Adjacency ‚ 2 s ℓ s h = 훽 퐴 푥 + 푝푎푡ℎ 푠푐표푟푒 훽 = 훽 푠 퐴 , 푦 + ⋯ qr qr qr ℓ ƒ . § ℓ over ALL paths of length Sum exponentially 1 a parameter of the predictor, is < 훽 < 0 § more heavily to count short damped paths predictions like common neighbors § Small 훽 = much : § Two forms authors collaborated, 0 1 • : Unweighted if two otherwise the collaboration • Weighted : strength of form for the entire score matrix: Closed ‡ . 퐼 훽퐴 퐼 − − GRAPH MINING WS 2016 20

21 LP Methods: Path based § 퐺 Consider a random walk on that starts at x and iteratively 표푙푑 moves to a neighbor of x chosen uniformly random from ( 푥 ) Γ The Hitting Time 퐻 of § from x to y is the expected number r , q steps it takes for the random walk starting at x to reach y. 푥 , 푠푐표푟푒 ) ( = − 퐻 푦 q , r § Commute Time from x to y is the expected number of steps The to travel from x to y and from y to x 푠푐표푟푒 ( 푥 , 푦 ) = − ( 퐻 ) 퐻 + r , r , q q Not symmetric, can be shown GRAPH MINING WS 2016 21

22 LP Methods: Path based § The hitting time and commute time measures are sensitive to - > periodically parts of the graph far away from x and y jump back to x Random walk on G § x and has a probability c of that starts at old step returning to x at each Random walk with restart § : Starts from x, with probability ) 1 – ( moves to a random neighbor and with probability 푐 푐 returns to x ‡ . ‡ . 퐼 − 푐퐷 1 푠 퐴 − 푐 푒 = q where 푠 is a similarity vector between x and all the other nodes in the graph and is the vector that has all 0, but a 1 in position x 푒 q 푠푐표푟푒 푥 , 푦 = 푠 r GRAPH MINING WS 2016 22

23 Path based: approaches SimRank Intuition: objects are similar if they are referenced by similar objects two 2 2 2 G(V,E) ,E ) (V G Structural context SimRank . context similarity - : a measure of structural 2002 and Jennifer . SIGKDD, Glen Jeh Widom GRAPH MINING WS 2016 23

24 Path based: SimRank approaches Expected Meeting : how soon two random Distance (EMD) surfers are expected to meet at the same node if they started at nodes x and y and randomly walked (in lock step) the graph backwards ∞ = ) ⋅ , ⋅ ( 푠푐표푟푒 • ¥ ) = u,w score( • score(u , v) = • 푠푐표푟푒 ( ⋅ , ⋅ ) 3 = ) = 1 => no node will meet • score(v, w => any two node will meet in and w are much more => v expectedly 3 steps, the similarity is than similar u is to v or w. lower than the previous for v,w GRAPH MINING WS 2016 24

25 Path based: SimRank approaches 2 Let us consider G § c a b ) as a state of the tour in G: if A node ( moves to , , b moves § a 2 in G, then ( a , b ) moves to ( c , d ) in G to d 2 of length n represents a pair of tours in G where each has length n A tour in G 2 What are the states in G § that correspond to “meeting” points? Singleton nodes (common neighbors) , a § The EMD m( b ) is just the expected distance (hitting time) in 2 , ) and any singleton node a between ( G b b ) and end at § The sum is taken over all walks that start from ( a , a singleton node GRAPH MINING WS 2016 25

26 LP Methods: Low Rank Approximations Assume that a small number of latent factors describe the social § and attribute link strength § Take the adjacency matrix A and a parameter r § Extract these r latent factors using a low rank matrix approximations § Apply SVD to find a factorization of A § Take the r that best approximates A GRAPH MINING WS 2016 26

27 Singular Value Decomposition ! Diagonal matrix σ v ù é ù é 1 1 ! ê ú ú ê σ ! ! ! v 2 2 T ê ú ê ú ] [ U = = Σ V u u u $ A r 2 1 ê ú ú ê # " r ] n [ × [ [ ] r × r n ] r × ! ê ú ê ú σ v r û ë r û ë Orthonormal Orthonormal matrix matrix § A : rank of matrix r T T ≥ 흈 : singular values (square roots of eig - vals § AA 흈 ≥ , A ⋯ A ≥ ) 흈 ퟐ ퟏ 풓 T : ) , ... , 푢 푢 , left singular vectors ( eig - vectors of § AA 푢 F s . T 푣 푣 § vectors of , ... , right singular vectors ( : A A 푣 , - eig ) . s F ¥ ¥ ¥ + 푣 휎 푢 푢 푢 푣 퐴 푣 + ... + 휎 = 휎 s F F . s F . . s GRAPH MINING WS 2016 27

28 LP Methods: Unseen bigrams Intuition « o compute the score( x,y ) use top - k nodes 푆 T that are similar to x using any q of the previous scores and intersect the neighbors of y ∗ « 푆 푠푐표푟 푧 푦 푒 Γ § ∈ : 푧 ∩ = 푦 , 푥 q ¦;=

29 LP Methods: Clustering Intuition Improve the score deleting the ”weakest” edges 푠푐표푟푒 푥 , 푦 Compute 퐸 § for all edges in 123 푝 delete 1 − 푝 fraction of § Given a user defined parameter the edges whose score is the lowest Recompute 푠푐표푟푒 푥 , 푦 for all pairs in the subgraph § GRAPH MINING WS 2016 29

30 Evaluation of Link Prediction § p outputs a ranked list L Each link predictor of pairs in p 푉 × 푉 ∖ 퐸 : predicted new collaborations in decreasing order 표푙푑 of confidence § If you have defined a core then consider ∗ ∗ = = 퐸 퐸 ∩ 퐶표푟푒 × 퐶표푟푒 | | 퐸 ;<= ;<= ;<= Evaluation method: Size of the intersection of × 퐿 , and 퐶표푟푒 § the first n edge predictions from 퐶표푟푒 that are in 푝 ∗ § 퐸 the set ;<= n predictions are correct (precision?) How many of the (relevant) top - GRAPH MINING WS 2016 30

31 Evaluation of LP: baseline § random predictor Baseline: Randomly select pairs of nodes who are not connected in the § training interval § Probability that a random prediction is correct 퐸 ;<= | ±1F< | − | 퐸 | 123 s GRAPH MINING WS 2016 31

32 Evaluation: improvement over random GRAPH MINING WS 2016 32

33 Evaluation: improvement over random GRAPH MINING WS 2016 33

34 Evaluation: Average relevance performance (random) average ratio the five § over the predictor's of datasets given performance versus a baseline . predictor's performance error the indicate the § bars maximum of this minimum and over ratio five datasets . the § parameters for the starred the are weighted ( 1 ) for predictors : ( β Katz, . 005 ; = 2 ) for Katz 0 clustering, β 1 = 0 . 001 ; ρ = 0 . 15 ; rank β 0 . 1 ; ( 3 ) for low - = inner 2 product, rank = 256 ; ( 4 ) for rooted Pagerank , α = 0 . 15 ; ( 5 ) for bigrams, unseen unweighted, common neighbors with δ = 8 ; and ( 6 ) for 0 SimRank C ( γ) = , . 8 . GRAPH MINING WS 2016 34

35 Evaluation: prediction overlap How similar are the predictions made by the Why? different methods? of common Number predictions correct GRAPH MINING WS 2016 35

36 Unsupervised Link Prediction Challenges § - world effect Shortest paths suffer of the small § - qc) Improve performance. Even the best (Katz clustering on gr correct on only about 16% of its prediction § Improve efficiency on very large networks (approximation of distances) Consider time effect : most recent links are more important § § Exploit additional information (attributes, text, ... ) GRAPH MINING WS 2016 36

37 Lecture road Unsupervised methods Classification approaches Who to follow GRAPH MINING WS 2016 37

38 Classification for link prediction Intuition Use any supervised classifier to predict if a link exists (=1) or no (=0) As features use weak node to node link predictors (e.g., common neighbors) : special random walk stopping at size or when cycle PropFlow l GRAPH MINING WS 2016 38

39 Why Supervised learning for LP? Restricted to n neighbors - (just look at the links from one node to nodes at distance n § Unsupervised methods like those that we have seen so far might work well with some network but do not generalize to others Features are dependent § GRAPH MINING WS 2016 39

40 How to get training data? length (in time) of computing features § 휏 q 휏 length of determining the class attribute § r § Large 휏 ⇒ better quality of features as the network reaches q saturation § Increasing 휏 ⇒ increases labeled data and final performance r GRAPH MINING WS 2016 40

41 Metrics for Performance Evaluation § Confusion Matrix: contains the number of T • rue positive: correctly predicted links that are actually links True negative: number of correctly predicted non - links • • False positive: number of predicted links that are not links • False negative: number of non - predicted links that are actually links PREDICTED CLASS Class=Yes Class=No Class=Yes TP FN ACTUAL FP Class=No TN CLASS TN TP + Accuracy = FN FP + + + TN TP GRAPH MINING WS 2016 41

42 ROC stands for Receiver Operating Characteristic ROC Curve Show the performance of a binary § classifier • TPR (sensitivity)=TP/(TP+FN) (how many data points are correctly classified among those that are actually positive) FPR = FP/(TN+FP) (percentage of negative • classified as positive) (0,0): declare everything § to be negative class § (1,1): declare everything to be positive class (0,1): ideal § Diagonal line: Random guessing § § Below diagonal line: prediction is worse than random AUC : area under the ROC curve GRAPH MINING WS 2016 42

43 Drawing ROC curve a Pairs of nodes ordered by score (parameter k) Number of correctly detected links if only considered the first k pairs Number of links that are Number of links not links - Number of non not link recognized in the correctly detected detected k - top GRAPH MINING WS 2016 43

44 ROC curve: an example § Assume that you have 4 nodes => 6 pairs § Order the pairs by decreasing score § Mark if the predicted link at that threshold is actually a link § Compute TP, TN, FP, FN, TPR, FPR (3,4) (2,3) (1,2) (1,4) (2,4) (1,3) 1.2 0.75 0.9 Score 0.6 0.5 0.2 0.4 1 No Yes No Yes Yes Actual No Link 0.8 1 3 TP 3 2 2 3 0.6 2 0 2 3 3 TN 1 1 2 3 1 FP 0 0 0.4 FN 2 1 1 0 0 0 True positive rate 0.2 1 2/3 1 1 TPR 1/3 2/3 0 FPR 1 2/3 1/3 1/3 0 0 0.4 0 0.2 1.2 0.6 0.8 1 False positive rate GRAPH MINING WS 2016 44

45 Results GRAPH MINING WS 2016 45

46 Lecture road Unsupervised methods Classification approaches Who to follow GRAPH MINING WS 2016 46

47 Who to Follow § (“Who to Follow"): the Twitter user recommendation service TwitWtf § users, 500 million tweets every day (2016) 317 million http ://www.internetlivestats.com/twitter - statistics/ § Twitter needs to help existing and new users to discover connections to sustain and grow § Also used for search relevance, discovery, promoted products, etc. GRAPH MINING WS 2016 47

48 The Twitter graph § follows Node: user (directed) edge: Statistics (August 2012) § over 20 billion edges (only active users) • power law distributions of in • degrees and out - degrees. - • over 1000 with more than 1 million followers, • 25 users with more than 10 million followers. - http://blog.ouseful.info/2011/07/07/visualising - twitter - friend - connections - using - gephi an - example - using - wireduk - friends - network/ GRAPH MINING WS 2016 48

49 Algorithms: Circle of trust Circle trust : the result of an egocentric random walk of to (rooted) PageRank ) (similar personalized Computed in an online fashion (from scratch each § given a time) set parameters (# of of walk steps, reset probability, random pruning settings to discard low probability vertices, parameters large to of outgoing edges at vertices with sampling out - control degrees, etc . ) § Used in a variety of Twitter products , e . g . , in search and circle discovery, from users in one's content of trust upweighted GRAPH MINING WS 2016 49

50 Algorithms Asymmetric nature of the follow relationship § (other social networks e.g., Facebook or LinkedIn require the consent of both participating members) § - irected edge case is similar to the user D item recommendations problem where the “item” is also a user. GRAPH MINING WS 2016 50

51 Algorithms: SALSA (Stochastic Approach for Link - Structure Analysis) SALSA a variation of HITS As in HITS hubs authorities HITS good § authorities Good hubs point to § Good authorities are pointed by good hubs sum of the authority weights of the hub weight = authorities pointed to by the hub h a = å j i i j : j ® that sum of the hub weights = authority weight . point to this authority a = h authorities hubs å j i j ® : i j GRAPH MINING WS 2016 51

52 In the next episode ... Student presentations Community detection And much more ... GRAPH MINING WS 2016 52

53 Questions? GRAPH MINING WS 2016 53

54 References § , R.N., Lussier , J.T. and Chawla, N.V.. New perspectives and methods in Lichtenwalter . link prediction KDD, 2010. Lü , L. and Zhou, T., 2011. Link prediction in complex networks: A survey . Physica § A: Statistical Mechanics and its Applications , 390 (6), pp.1150 - 1170 . § Liben-Nowell , D. and Kleinberg, J., 2007. The link-prediction problem for social networ Journal of the American society for information science and ks. technology , 58 (7), pp.1019 - 1031 . § Gupta, P., Goel , A., Lin, J., Sharma, A., Wang, D. and Zadeh , R. Wtf: The who to follow service at twitter. WWW, 2013. GRAPH MINING WS 2016 54

Related documents

MCO 1200.17E MILITARY OCCUPATIONAL SPECIALTIES MANUAL (SHORT TITLE: MOS MANUAL)

MCO 1200.17E MILITARY OCCUPATIONAL SPECIALTIES MANUAL (SHORT TITLE: MOS MANUAL)

DEPAR T MENT THE NAVY OF ADQ UARTE UNI T ED ST ATE S MAR INE CORPS HE RS RINE COR N PS PENT 3000 MA AGO 20350-3000 NGTON, HI D.C. W AS 7E 00 .1 12 MCO 465 c AUG 0 8 013 2 ORDER 1200.17E MARINE CORPS C...

More info »
Fourth National Report on Human Exposure to Environmental Chemicals Update

Fourth National Report on Human Exposure to Environmental Chemicals Update

201 8 Fourth National Report on Human Exposure to Environmental Chemicals U pdated Tables, March 2018 , Volume One

More info »
AT Commands Reference Guide

AT Commands Reference Guide

AT Commands Reference Guide GE863-QUAD, GE863-PY, GE863-GPS, GM862-QUAD, GM862-QUAD-PY, GE862-GPS, GE864-QUAD, GE864-PY, GC864-QUAD and GC864-PY 80000ST10025a Rev. 0 - 04/08/06

More info »
JO 7400.11C   Airspace Designations and Reporting Points

JO 7400.11C Airspace Designations and Reporting Points

U.S. DEPARTMENT OF TRANSPORTATION ORDER FEDERAL AVIATION ADMINISTRATION 7400.11C JO Air Traffic Organization Policy August 13, 2018 SUBJ: Airspace Designations and Reporting Points . This O rder, publ...

More info »
catalog 2019

catalog 2019

2019 ® HARLEY-DAVIDSON GENUINE MOTOR PARTS & ACCESSORIES

More info »
CityNT2019TentRoll 1

CityNT2019TentRoll 1

STATE OF NEW YORK 2 0 1 9 T E N T A T I V E A S S E S S M E N T R O L L PAGE 1 VALUATION DATE-JUL 01, 2018 COUNTY - Niagara T A X A B L E SECTION OF THE ROLL - 1 CITY - North Tonawanda TAX MAP NUMBER ...

More info »
RIE Tenant List By Docket Number

RIE Tenant List By Docket Number

SCRIE TENANTS LIST ~ By Docket Number ~ Borough of Bronx SCRIE in the last year; it includes tenants that have a lease expiration date equal or who have received • This report displays information on ...

More info »
OperatorHoursReport

OperatorHoursReport

John Bel Edwards Rebekah E. Gee MD, MPH SECRETARY GOVERNOR State of Louisiana Louisiana Department of Health Office of Public Health Certified Water and Wastewater Operators 2018 - 2019 Hours Hours li...

More info »
32 41.indd

32 41.indd

Assembly Tools Production Fastening and Drilling 1-5.indd 1 1-5.indd 1 7/2/08 12:52:55 PM 7/2/08 12:52:55 PM

More info »
DoD7045.7H

DoD7045.7H

DoD 7045.7-H EPARTMENT OF D EFENSE D F UTURE Y EARS D EFENSE P ROGRAM (FYDP) S TRUCTURE Codes and Definitions for All DoD Components Office of the Director, Program Analysis and Evaluation A pril 2004

More info »
WEF GlobalInformationTechnology Report 2014

WEF GlobalInformationTechnology Report 2014

Insight Report The Global Information Technology Report 2014 Rewards and Risks of Big Data Beñat Bilbao-Osorio, Soumitra Dutta, and Bruno Lanvin, Editors

More info »
OCS Operations Field Directory

OCS Operations Field Directory

Gulf of Mexico OCS Region OCS Operations Field Directory (Includes all active and expired fields and leases) Quarterly Repor t, as of March 31 , 201 9 U.S. Department of the Interior Bureau of Ocean E...

More info »
sustainability path design guidelines

sustainability path design guidelines

First Last Mile Strategic Plan & PLANNING GUIDELINES Sounds good, I haven’t been to LACMA in a while...the Pathway? Hmm...I’ll check it out. See you soon! The Meet-Up! The Meet-Up! In sunny downtown L...

More info »
CompleteBusBook

CompleteBusBook

February 10, 2019 $ 1 BUS BOOK EFFECTIVE THROUGH JUNE 8, 2019 OCBus.com EFECTIVO HASTA EL 8 DE JUNIO 2019 EASY JUST GOT EASIER. Upgrade to version 2.0 See back for cool new features! CHANGE HIGHLIGHTS...

More info »
UNSCEAR 2008 Report Vol.I

UNSCEAR 2008 Report Vol.I

This publication contains: VOLUME I: SOURCES SOURCES AND EFFECTS Report of the United Nations Scientific Committee on the Effects of Atomic Radiation to the General Assembly OF IONIZING RADIATION Scie...

More info »
C:\JIM\PLUMBOOK\2012PL~1\LIVE\76304.002

C:\JIM\PLUMBOOK\2012PL~1\LIVE\76304.002

[COMMITTEE PRINT] UNITED STATES GOVERNMENT Policy and Supporting Positions f Committee on Oversight and Government Reform U.S. House of Representatives 112th Congress, 2d Session DECEMBER 1, 2012 Avai...

More info »
vol9 organic ligands

vol9 organic ligands

C HERMODYNAMICS HEMICAL T OMPOUNDS AND C OMPLEXES OF OF C U, Np, Pu, Am, Tc, Se, Ni and Zr O ELECTED WITH RGANIC L IGANDS S Wolfgang Hummel (Chairman) Laboratory for Waste Management Paul Scherrer Ins...

More info »
Administrative Work in the Information Technology Group, GS 2200

Administrative Work in the Information Technology Group, GS 2200

Administrative Work in the Information Issued: May 2001 Revised: 8/03, 9/08, 5/11, October 2018 Technology Group, 2200 Job Family Standard for Administrative Work in the Information Technology Group, ...

More info »