Identifying useful learnwares via learnable specification
Zhi-Yu SHEN1, Ming LI1,2()
. National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China . School of Artificial Intelligence, Nanjing University, Nanjing 210023, China
The learnware paradigm has been proposed as a new manner for reusing models from a market of various well-trained models, which can relieve users’ burden of training a new model from scratch. A learnware consists of a well-trained model and a specification which explains the purpose or specialty of the model without revealing data. By specification matching, the market can identify the most useful learnwares for users’ tasks. Prior art attempted to generate the specification by a reduced kernel mean embedding approach. However, such kind of specification is defined by some pre-designed kernel function, which lacks flexibility. In this paper, we advance a methodology for direct specification learning from data, introducing a novel neural network named SpecNet for this purpose. Our approach accepts unordered datasets as input and subsequently produces specification vectors in a latent space. Notably, the flexibility and efficiency of our learned specifications are underscored by their derivation from diverse tasks, rendering them particularly adept for learnware identification. Empirical studies provide validation for the efficacy of our proposed approach.
Z H Zhou . Learnware: on the future of machine learning. Frontiers of Computer Science, 2016, 10( 4): 589–590
2
P, Tan Z H, Tan Y, Jiang Z H Zhou . Towards enabling learnware to handle heterogeneous feature spaces. Machine Learning, 2024, 113( 4): 1839–1860
3
Y J, Zhang Y H, Yan P, Zhao Z H Zhou . Towards enabling learnware to handle unseen jobs. In: Proceedings of the 35th AAAI Conference on Artificial Intelligence. 2021, 10964−10972
4
L Z, Guo Z, Zhou Y F, Li Z H Zhou . Identifying useful learnwares for heterogeneous label spaces. In: Proceedings of the 40th International Conference on Machine Learning. 2023, 12122−12131
5
J, Hoffman E, Tzeng T, Park J Y, Zhu P, Isola K, Saenko A, Efros T Darrell . CyCADA: cycle-consistent adversarial domain adaptation. In: Proceedings of the 35th International Conference on Machine Learning. 2018, 1994−2003
6
S, Ben-David J, Blitzer K, Crammer A, Kulesza F, Pereira J W Vaughan . A theory of learning from different domains. Machine Learning, 2010, 79(1−2): 151−175
7
Y, Ganin V Lempitsky . Unsupervised domain adaptation by backpropagation. In: Proceedings of the 32nd International Conference on Machine Learning. 2015, 1180−1189
8
A, Kumar T, Ma P Liang . Understanding self-training for gradual domain adaptation. In: Proceedings of the 37th International Conference on Machine Learning. 2020, 507
9
Y, Luo Z, Wang Z, Huang M Baktashmotlagh . Progressive graph learning for open-set domain adaptation. In: Proceedings of the 37th International Conference on Machine Learning. 2020, 600
10
X Z, Wu W, Xu S, Liu Z H Zhou . Model reuse with reduced kernel mean embedding specification. IEEE Transactions on Knowledge and Data Engineering, 2023, 35( 1): 699–710
11
A, Smola A, Gretton L, Song B Schölkopf . A hilbert space embedding for distributions. In: Proceedings of the 18th International Conference on Algorithmic Learning Theory. 2007, 13−31
12
Y X, Ding Z H Zhou . Boosting-based reliable model reuse. In: Proceedings of the 12th Asian Conference on Machine Learning. 2020, 145−160
13
S, Ben-David J, Blitzer K, Crammer F Pereira . Analysis of representations for domain adaptation. In: Proceedings of the 19th International Conference on Neural Information Processing Systems. 2006, 137−144
14
P, Germain A, Habrard F, Laviolette E Morvant . A new PAC-Bayesian perspective on domain adaptation. In: Proceedings of the 33rd International Conference on Machine Learning. 2016, 859−868
15
M Y, Liu O Tuzel . Coupled generative adversarial networks. In: Proceedings of the 30th International Conference on Neural Information Processing Systems. 2016, 469−477
16
W, Chang Y, Shi H D, Tuan J Wang . Unified optimal transport framework for universal domain adaptation. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. 2022, 2140
17
Y, Zhu X, Wu J, Qiang Y, Yuan Y Li . Representation learning via an integrated autoencoder for unsupervised domain adaptation. Frontiers of Computer Science, 2023, 17( 5): 175334
18
S J, Pan Q Yang . A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 2010, 22( 10): 1345–1359
19
Y X, Ding X Z, Wu K, Zhou Z H Zhou . Pre-trained model reusability evaluation for small-data transfer learning. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. 2022, 2710
20
B, Wang J A, Mendez M B, Cai E Eaton . Transfer learning via minimizing the performance gap between domains. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019, 955
21
N, Houlsby A, Giurgiu S, Jastrzebski B, Morrone Laroussilhe Q, De A, Gesmundo M, Attariyan S Gelly . Parameter-efficient transfer learning for NLP. In: Proceedings of the 36th International Conference on Machine Learning. 2019, 2790−2799
22
Y, Yang J, Guo G, Li L, Li W, Li J Yang . Alignment efficient image-sentence retrieval considering transferable cross-modal representation learning. Frontiers of Computer Science, 2024, 18( 1): 181335
23
Z, Wang Z, Dai B, Póczos J Carbonell . Characterizing and avoiding negative transfer. In: Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, 11293−11302
24
D, Maturana S Scherer . VoxNet: a 3D convolutional neural network for real-time object recognition. In: Proceedings of 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems. 2015, 922−928
25
C R, Qi H, Su M, Nießner A, Dai M, Yan L J Guibas . Volumetric and multi-view CNNs for object classification on 3D data. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. 2016, 5648−5656
26
Z, Wu S, Song A, Khosla F, Yu L, Zhang X, Tang J Xiao . 3D ShapeNets: a deep representation for volumetric shapes. In: Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. 2015, 1912−1920
27
O, Vinyals S, Bengio M Kudlur . Order matters: sequence to sequence for sets. In: Proceedings of the 4th International Conference on Learning Representations. 2016
28
Charles R, Qi H, Su M, Kaichun L J Guibas . PointNet: deep learning on point sets for 3D classification and segmentation. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. 2017, 77−85
29
Y, Le X Yang . Tiny ImageNet visual recognition challenge. In: Proceedings of the Stanford CS231N Convolutional Neural Networks for Visual Recognition. 2015
30
A Krizhevsky . Learning multiple layers of features from tiny images. Technical Report TR-2009. Toronto: University of Toronto, 2009
